Proactive process-level live migration in HPC environments

ABSTRACT

As the number of nodes in high-performance computing environments keeps increasing, faults are becoming common place. Reactive fault tolerance (FT) often does not scale due to massive I/O requirements and relies on manual job resubmission.

This work complements reactive with proactive FT at the process level. Through health monitoring, a subset of node failures can be anticipated when one's health deteriorates. A novel process-level live migration mechanism supports continued execution of applications during much of processes migration. This scheme is integrated into an MPI execution environment to transparently sustain health-inflicted node failures, which eradicates the need to restart and requeue MPI jobs. Experiments indicate that 1-6.5 seconds of prior warning are required to successfully trigger live process migration while similar operating system virtualization mechanisms require 13-24 seconds. This self-healing approach complements reactive FT by nearly cutting the number of checkpoints in half when 70% of the faults are handled proactively.

REFERENCES

• 1: "Advanced configuration & power interface," http://www.acpi.info.
• 2: "Readable dirty-bits for IA64 linux," https://www.gelato.unsw.edu.au/archives/gelato-technical/2005-November/001080.html.
• 3: R. T. Aulwes, D. J. Daniel, N. N. Desai, R. L. Graham, L. D. Risinger, M. A. Taylor, T. Woodall, and M. Sukalski, "Architecture of LA-MPI, a network-fault-tolerant MPI," in IPDPS, 2004.
• 4: A. Barak and R. Wheeler, "MOSIX: An integrated multiprocessor UNIX," in Proceedings of the Winter 1989 USENIX Conference. Berkeley, CA, USA: USENIX, 1989, pp. 101--112.
• 5: George Bosilca , Aurelien Bouteiller , Franck Cappello , Samir Djilali , Gilles Fedak , Cecile Germain , Thomas Herault , Pierre Lemarinier , Oleg Lodygensky , Frederic Magniette , Vincent Neri , Anton Selikhov, MPICH-V: toward a scalable fault tolerant MPI for volatile nodes, Proceedings of the 2002 ACM/IEEE conference on Supercomputing, p.1-18, November 16, 2002, Baltimore, Maryland
• 6: Jiannong Cao , Yinghao Li , Minyi Guo, Process Migration for MPI Applications based on Coordinated Checkpoint, Proceedings of the 11th International Conference on Parallel and Distributed Systems (ICPADS'05), p.306-312, July 20-22, 2005 [doi>10.1109/ICPADS.2005.241]
• 7: S. Chakravorty, C. Mendes, and L. Kale, "Proactive fault tolerance in MPI applications via task migration," in HiPC, 2006.
• 8: S. Chakravorty, C. Mendes, and L. Kale, "Proactive fault tolerance in large systems," in HPCRI: 1st Workshop on High Performance Computing Reliability Issues, in Proceedings of HPCA-11, 2005.
• 9: S. Chakravorty, C. Mendes, and L. Kale, "A fault tolerance protocol with fast fault recovery," in IPDPS, 2007.
• 10: Christopher Clark , Keir Fraser , Steven Hand , Jacob Gorm Hansen , Eric Jul , Christian Limpach , Ian Pratt , Andrew Warfield, Live migration of virtual machines, Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation, p.273-286, May 02-04, 2005
• 11: Fred Douglis , John Ousterhout, Transparent process migration: design alternatives and the sprite implementation, Software—Practice & Experience, v.21 n.8, p.757-785, Aug. 1991 [doi>10.1002/spe.4380210802]
• 12: Cong Du , Xian-He Sun Sun, MPI-Mitten: Enabling Migration Technology in MPI, Proceedings of the Sixth IEEE International Symposium on Cluster Computing and the Grid, p.11-18, May 16-19, 2006 [doi>10.1109/CCGRID.2006.71]
• 13: C. Du, X.-H. Sun, and K. Chanchio, "HPCM: A pre-compiler aided middleware for the mobility of legacy code," in IEEE Cluster, 2003.
• 14: Cong Du , Xian-He Sun , Ming Wu, Dynamic Scheduling with Process Migration, Proceedings of the Seventh IEEE International Symposium on Cluster Computing and the Grid, p.92-99, May 14-17, 2007 [doi>10.1109/CCGRID.2007.46]
• 15: J. Duell, "The design and implementation of berkeley lab's linux checkpoint/restart," Lawrence Berkeley National Laboratory, TR, 2000.
• 16: Elmootazbellah N. Elnozahy , Willy Zwaenepoel, Manetho: Transparent Roll Back-Recovery with Low Overhead, Limited Rollback, and Fast Output Commit, IEEE Transactions on Computers, v.41 n.5, p.526-531, May 1992 [doi>10.1109/12.142678]
• 17: Graham E. Fagg , Jack Dongarra, FT-MPI: Fault Tolerant MPI, Supporting Dynamic Applications in a Dynamic World, Proceedings of the 7th European PVM/MPI Users' Group Meeting on Recent Advances in Parallel Virtual Machine and Message Passing Interface, p.346-353, January 2000
• 18: Sanjay Ghemawat , Howard Gobioff , Shun-Tak Leung, The Google file system, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 19: Roberto Gioiosa , Jose Carlos Sancho , Song Jiang , Fabrizio Petrini, Transparent, Incremental Checkpointing at Kernel Level: a Foundation for Fault Tolerance for Parallel Computers, Proceedings of the 2005 ACM/IEEE conference on Supercomputing, p.9, November 12-18, 2005 [doi>10.1109/SC.2005.76]
• 20: Xiaohui Gu , Spiros Papadimitriou , Philip S. Yu , Shu-Ping Chang, Toward Predictive Failure Management for Distributed Stream Processing Systems, Proceedings of the 2008 The 28th International Conference on Distributed Computing Systems, p.825-832, June 17-20, 2008 [doi>10.1109/ICDCS.2008.34]
• 21: Prashasta Gujrati , Yawei Li , Zhiling Lan , Rajeev Thakur , John White, A Meta-Learning Failure Predictor for Blue Gene/L Systems, Proceedings of the 2007 International Conference on Parallel Processing, p.40, September 10-14, 2007 [doi>10.1109/ICPP.2007.9]
• 22: Chung-hsing Hsu , Wu-chun Feng, A Power-Aware Run-Time System for High-Performance Computing, Proceedings of the 2005 ACM/IEEE conference on Supercomputing, p.1, November 12-18, 2005 [doi>10.1109/SC.2005.3]
• 23: Mohammad Tanvir Huda , Heinz W. Schmidt , Ian D. Peake, An Agent Oriented Proactive Fault-Tolerant Framework for Grid Computing, Proceedings of the First International Conference on e-Science and Grid Computing, p.304-311, December 05-08, 2005 [doi>10.1109/E-SCIENCE.2005.15]
• 24: J. Hursey, J. M. Squyres, and A. Lumsdaine, "A checkpoint and restart service specification for Open MPI," Indiana University, Computer Science Department, Technical Report, 2006.
• 25: J. Hursey, J. M. Squyres, T. I. Mattox, and A. Lumsdaine, "The design and implementation of checkpoint/restart process fault tolerance for Open MPI," in DPDNS, Mar. 2007.
• 26: Eric Jul , Henry Levy , Norman Hutchinson , Andrew Black, Fine-grained mobility in the Emerald system, ACM Transactions on Computer Systems (TOCS), v.6 n.1, p.109-133, Feb. 1988 [doi>10.1145/35037.42182]
• 27: Yawei Li , Prashasta Gujrati , Zhiling Lan , Xian-he Sun, Fault-Driven Re-Scheduling For Improving System-level Fault Resilience, Proceedings of the 2007 International Conference on Parallel Processing, p.39, September 10-14, 2007 [doi>10.1109/ICPP.2007.42]
• 28: M. Litzkow, "Remote unix - turning idle workstations into cycle servers," in Usenix Summer Conference, 1987, pp. 381--384.
• 29: M. Litzkow, T. Tannenbaum, J. Basney, and M. Livny, "Checkpoint and migration of UNIX processes in the Condor distributed processing system," University of Wisconsin - Madison Computer Sciences Department, Tech. Rep. UW-CS-TR-1346, April 1997.
• 30: Jiuxing Liu , Wei Huang , Bulent Abali , Dhabaleswar K. Panda, High performance VMM-bypass I/O in virtual machines, Proceedings of the annual conference on USENIX '06 Annual Technical Conference, p.3-3, May 30-June 03, 2006, Boston, MA
• 31: Aravind Menon , Alan L. Cox , Willy Zwaenepoel, Optimizing network virtualization in Xen, Proceedings of the annual conference on USENIX '06 Annual Technical Conference, p.2-2, May 30-June 03, 2006, Boston, MA
• 32: Dejan S. Milojičić , Fred Douglis , Yves Paindaveine , Richard Wheeler , Songnian Zhou, Process migration, ACM Computing Surveys (CSUR), v.32 n.3, p.241-299, Sept. 2000 [doi>10.1145/367701.367728]
• 33: Arun Babu Nagarajan , Frank Mueller , Christian Engelmann , Stephen L. Scott, Proactive fault tolerance for HPC with Xen virtualization, Proceedings of the 21st annual international conference on Supercomputing, June 17-21, 2007, Seattle, Washington [doi>10.1145/1274971.1274978]
• 34: A. Oliner, R. Sahoo, J. Moreira, M. Gupta, and A. Sivasubramaniam, "Fault-aware job scheduling for BlueGene/L systems," in IPDPS, 2004.
• 35: I. Philp, "Software failures and the road to a petaflop machine," in HPCRI: 1st Workshop on High Performance Computing Reliability Issues, in Proceedings of HPCA-11. IEEE Computer Society, 2005.
• 36: James S. Plank , Micah Beck , Gerry Kingsley , Kai Li, Libckpt: transparent checkpointing under Unix, Proceedings of the USENIX 1995 Technical Conference Proceedings on USENIX 1995 Technical Conference Proceedings, p.18-18, January 16-20, 1995, New Orleans, Louisiana
• 37: Michael L. Powell , Barton P. Miller, Process migration in DEMOS/MP, Proceedings of the ninth ACM symposium on Operating systems principles, p.110-119, October 10-13, 1983, Bretton Woods, New Hampshire, United States
• 38: Heather Quinn , Paul Graham, Terrestrial-Based Radiation Upsets: A Cautionary Tale, Proceedings of the 13th Annual IEEE Symposium on Field-Programmable Custom Computing Machines, p.193-202, April 18-20, 2005 [doi>10.1109/FCCM.2005.61]
• 39: S. Rani, C. Leangsuksun, A. Tikotekar, V. Rampure, and S. Scott, "Toward efficient failre detection and recovery in HPC," in High Availability and Performance Computing Workshop, 2006.
• 40: R. K. Sahoo , A. J. Oliner , I. Rish , M. Gupta , J. E. Moreira , S. Ma , R. Vilalta , A. Sivasubramaniam, Critical event prediction for proactive management in large-scale computer clusters, Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, August 24-27, 2003, Washington, D.C. [doi>10.1145/956750.956799]
• 41: S. Sankaran, J. M. Squyres, B. Barrett, A. Lumsdaine, J. Duell, P. Hargrove, and E. Roman, "The LAM/MPI checkpoint/restart framework: System-initiated checkpointing," in LACSI, Oct. 2003.
• 42: Hertong Song , Chokchai box Leangsuksun , Raja Nassar, Availability Modeling and Analysis on High Performance Cluster Computing Systems, Proceedings of the First International Conference on Availability, Reliability and Security, p.305-313, April 20-22, 2006 [doi>10.1109/ARES.2006.37]
• 43: J. M. Squyres and A. Lumsdaine, "A Component Architecture for LAM/MPI," in European PVM/MPI Users' Group Meeting, ser. Lecture Notes in Computer Science, no. 2840. Venice, Italy: Springer-Verlag, September / October 2003, pp. 379--387.
• 44: Georg Stellner, CoCheck: Checkpointing and Process Migration for MPI, Proceedings of the 10th International Parallel Processing Symposium, p.526-531, April 15-19, 1996
• 45: X.-H. Sun, Z. Lan, Y. Li, H. Jin, and Z. Zheng, "Towards a fault-aware computing environment," in HAPCW, Mar. 2008.
• 46: Marvin M. Theimer , Keith A. Lantz , David R. Cheriton, Preemptable remote execution facilities for the V-system, Proceedings of the tenth ACM symposium on Operating systems principles, p.2-12, December 1985, Orcas Island, Washington, United States
• 47: A. Tikotekar, C. Leangsuksun, and S. L. Scott, "On the survivability of standard MPI applications," in LCI International Conference on Linux Clusters: The HPC Revolution, May 2006.
• 48: A. Tikotekar, G. Vallée, T. Naughton, S. L. Scott, and C. Leangsuksun, "Evaluation of fault-tolerant policies using simulation," in IEEE Cluster, Sep. 17--20, 2007.
• 49: Geoffroy Vallee , Renaud Lottiaux , David Margery , Christine Morin, Ghost Process: a Sound Basis to Implement Process Duplication, Migration and Checkpoint/Restart in Linux Clusters, Proceedings of the The 4th International Symposium on Parallel and Distributed Computing, p.97-104, July 04-06, 2005 [doi>10.1109/ISPDC.2005.29]
• 50: Geoffroy Vallee , Kulathep Charoenpornwattana , Christian Engelmann , Anand Tikotekar , Chokchai Leangsuksun , Thomas Naughton , Stephen L. Scott, A Framework for Proactive Fault Tolerance, Proceedings of the 2008 Third International Conference on Availability, Reliability and Security, p.659-664, March 04-07, 2008 [doi>10.1109/ARES.2008.171]
• 51: C. Wang, F. Mueller, C. Engelmann, and S. Scott, "A job pause service under LAM/MPI+BLCR for transparent fault tolerance," in IPDPS, Apr. 2007.
• 52: Frederick C. Wong , Richard P. Martin , Remzi H. Arpaci-Dusseau , David E. Culler, Architectural requirements and scalability of the NAS parallel benchmarks, Proceedings of the 1999 ACM/IEEE conference on Supercomputing (CDROM), p.41-es, November 14-19, 1999, Portland, Oregon, United States [doi>10.1145/331532.331573]
• 53: John W. Young, A first order approximation to the optimum checkpoint interval, Communications of the ACM, v.17 n.9, p.530-531, Sept. 1974 [doi>10.1145/361147.361115]

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Fault tolerance

Additional Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.4 Distributed Systems
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Performance attributes; Reliability, availability, and serviceability

General Terms:
Design, Experimentation, Measurement, Performance, Reliability