Scalable transparent checkpoint-restart of global address space applications on virtual machines over infiniband

ABSTRACT

Checkpoint-Restart is one of the most used software approaches to achieve fault-tolerance in high-end clusters. While standard techniques typically focus on user-level solutions, the advent of virtualization software has enabled efficient and transparent system-level approaches. In this paper, we present a scalable transparent system-level solution to address fault-tolerance for applications based on global address space (GAS) programming models on Infiniband clusters. In addition to handling communication, the solution addresses transparent checkpoint of user-generated files. We exploit the support for the Infiniband network in the Xen virtual machine environment. We have developed a version of the Aggregate Remote Memory Copy Interface (ARMCI) one-sided communication library capable of suspending and resuming applications. We present efficient and scalable mechanisms to distribute checkpoint requests and to backup virtual machines memory images and file systems. We tested our approach in the context of NWChem, a popular computational chemistry suite. We demonstrated that NWChem can be executed, without any modification to the source code, on a virtualized 8-node cluster with very little overhead (below 3%). We observe that the total checkpoint time is limited by disk I/O. Finally, we measured system-size depended components of the checkpoint time on up to 1024 cores (128 nodes), demonstrating the scalability of our approach in medium/large-scale systems.

REFERENCES

• 1: D. Abramson, J. Jackson, S. Muthrasanallur,G. Neiger, G. Regnier, R. Sankaran, I. Schoinas,R. Uhlig, B. Vembu, and J. Wiegert. Intel Technology Journal (Intel) 10 (3): 179--192.
• 2: Adam Beguelin , Erik Seligman , Peter Stephan, Application level fault tolerance in heterogeneous networks of workstations, Journal of Parallel and Distributed Computing, v.43 n.2, p.147-155, June 15, 1997 [doi>10.1006/jpdc.1997.1338]
• 3: Anita Borg , Wolfgang Blau , Wolfgang Graetsch , Ferdinand Herrmann , Wolfgang Oberle, Fault tolerance under UNIX, ACM Transactions on Computer Systems (TOCS), v.7 n.1, p.1-24, Feb. 1989 [doi>10.1145/58564.58565]
• 4: S. Chakravorty and L. Kale. A Fault Tolerance Protocol for Fast Fault Recovery. In IEEE/ACM International Parallel and Distributed Processing Symposium (IPDPS 2007), Long Beach, CA, March 2007.
• 5: Ludmila Cherkasova , Rob Gardner, Measuring CPU overhead for I/O processing in the Xen virtual machine monitor, Proceedings of the annual conference on USENIX Annual Technical Conference, p.24-24, April 10-15, 2005, Anaheim, CA
• 6: Yuri Dotsenko , Cristian Coarfa , John Mellor-Crummey, A Multi-Platform Co-Array Fortran Compiler, Proceedings of the 13th International Conference on Parallel Architectures and Compilation Techniques, p.29-40, September 29-October 03, 2004 [doi>10.1109/PACT.2004.3]
• 7: Paul Barham , Boris Dragovic , Keir Fraser , Steven Hand , Tim Harris , Alex Ho , Rolf Neugebauer , Ian Pratt , Andrew Warfield, Xen and the art of virtualization, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 8: K. Fraser, S. Hand, R. Neugebauer, I. Pratt, A. Warfield, and M. Williamson. Safe hardware access with the Xen virtual machine monitor. In In Proceedings of the OASIS ASPLOS Workshop, 2004, 2004.
• 9: Qi Gao , Weikuan Yu , Wei Huang , Dhabaleswar K. Panda, Application-Transparent Checkpoint/Restart for MPI Programs over InfiniBand, Proceedings of the 2006 International Conference on Parallel Processing, p.471-478, August 14-18, 2006 [doi>10.1109/ICPP.2006.26]
• 10: A. Geist and C. Engelmann. Development of Naturally Fault Tolerant Algorithms for Computing on 100,000 Processors. Oak Ridge National Laboratory, 2002.
• 11: Roberto Gioiosa , Jose Carlos Sancho , Song Jiang , Fabrizio Petrini, Transparent, Incremental Checkpointing at Kernel Level: a Foundation for Fault Tolerance for Parallel Computers, Proceedings of the 2005 ACM/IEEE conference on Supercomputing, p.9, November 12-18, 2005 [doi>10.1109/SC.2005.76]
• 12: W. Huang, Q. Gao, J. Liu, and D. Panda. High performance virtual machine migration with rdma over modern interconnects. Cluster Computing, 2007 IEEE International Conference on, pages 11--20, Sept. 2007.
• 13: Wei Huang , Jiuxing Liu , Matthew Koop , Bulent Abali , Dhabaleswar Panda, Nomad: migrating OS-bypass networks in virtual machines, Proceedings of the 3rd international conference on Virtual execution environments, June 13-15, 2007, San Diego, California, USA [doi>10.1145/1254810.1254833]
• 14: A. Kivity, Y. Kamay, D. Laor, U. Lublin, and A. Liguori. kvm: the linux virtual machine monitor. In Ottawa Linux Symposium, pages 225--230, July 2007.
• 15: N.-B. C. Laboratory. MPI over InfiniBand Project.
• 16: C. J. Li and W. K. Fuch. CATCH - Compiler Assisted Techniques for Checkpointing. In Proceedings of the International Symposium on Fault Tolerant Computing, pages 74--81, June 1990.
• 17: Jiuxing Liu , Wei Huang , Bulent Abali , Dhabaleswar K. Panda, High performance VMM-bypass I/O in virtual machines, Proceedings of the annual conference on USENIX '06 Annual Technical Conference, p.3-3, May 30-June 03, 2006, Boston, MA
• 18: J. Liu, J. Wu, S. P. Kini, D. Buntinas, W. Yu, B. Chandrasekaran, R. M. Noronha, P. Wyckoff, and D. K. Panda. MPI over infiniband: Early experiences.Technical Report OSU-CISRC-10/02-TR25, Ohio State Univ., Aug 2003.
• 19: Jiuxing Liu , Jiesheng Wu , Dhabaleswar K. Panda, High performance RDMA-based MPI implementation over infiniBand, International Journal of Parallel Programming, v.32 n.3, p.167-198, June 2004 [doi>10.1023/B:IJPP.0000029272.69895.c1]
• 20: Aravind Menon , Jose Renato Santos , Yoshio Turner , G. (John) Janakiraman , Willy Zwaenepoel, Diagnosing performance overheads in the xen virtual machine environment, Proceedings of the 1st ACM/USENIX international conference on Virtual execution environments, June 11-12, 2005, Chicago, IL, USA [doi>10.1145/1064979.1064984]
• 21: Arun Babu Nagarajan , Frank Mueller , Christian Engelmann , Stephen L. Scott, Proactive fault tolerance for HPC with Xen virtualization, Proceedings of the 21st annual international conference on Supercomputing, June 17-21, 2007, Seattle, Washington [doi>10.1145/1274971.1274978]
• 22: Jaroslaw Nieplocha , Robert J. Harrison , Richard J. Littlefield, Global arrays: a nonuniform memory access programming model for high-performance computers, The Journal of Supercomputing, v.10 n.2, p.169-189, June 1996
• 23: J. Nieplocha , V. Tipparaju , M. Krishnan , D. K. Panda, High Performance Remote Memory Access Communication: The Armci Approach, International Journal of High Performance Computing Applications, v.20 n.2, p.233-253, May 2006 [doi>10.1177/1094342006064504]
• 24: K. Parzyszek, J. Nieplocha, and R. A. Kendall. Generalized Portable SHMEM Library for High Performance Computing. In M. Guizani and X. Shen, editors, IASTED Parallel and Distributed Computing and Systems, pages 401--406, 2000.
• 25: Milos Prvulovic , Zheng Zhang , Josep Torrellas, ReVive: cost-effective architectural support for rollback recovery in shared-memory multiprocessors, Proceedings of the 29th annual international symposium on Computer architecture, May 25-29, 2002, Anchorage, Alaska
• 26: J. Ruscio, M. Heffner, and S. Varadarajan. DejaVu: Transparent User-Level Checkpointing, Migration and Recovery for Distributed Systems. In Proc. of the Intl. Parallel and Distributed Processing Symposium (IPDPS 2007), March 2007.
• 27: T. P. Straatsma , J. A. McCammon, Load balancing of molecular dynamics simulation with NWChem, IBM Systems Journal, v.40 n.2, p.328-341, February 2001
• 28: C. Wang, F. Mueller, C. Engelmann, and S. Scott. A Job Pause Service under LAM/MPI + BLCR for Transparent Fault Tolerance. In IEEE/ACM International Parallel and Distributed Processing Symposium (IPDPS 2007), March 2007.
• 29: Dennis Zimmer, VMware Server and VMware Player. The way forward for Virtualization, BoD, 2006

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.5 Reliability
          Subjects: Checkpoint/restart

General Terms:
Design, Reliability

Keywords:
checkpoint-restart, global address space, infiniband, virtual machines