Transparent checkpoints of closed distributed systems in Emulab

ABSTRACT

Emulab is a testbed for networked and distributed systems experimentation. Two guiding principles of its design are realism and control of experimentation. There is an inherent tension between these goals, however, and in some aspects of the testbed's design, Emulab's implementers favored realism over control. Thus, Emulab provides wide-ranging control over an experiment's environment and initial conditions, but relatively little control over its execution--in particular, the ability to suspend, preempt, or replay the experiment.

We have extended Emulab with a new means of control over experiment execution: the ability to cleanly checkpoint the execution of the set of nodes and networks that comprise an experiment. Conventional checkpoint mechanisms can easily degrade the fidelity of experiment results as a consequence of checkpoint downtimes, overheads of background state saving, and unintended distributed checkpoint synchronization effects. In this paper we demonstrate a checkpointing technique that is transparent with respect to the execution of the system under test, almost completely concealing the underlying checkpoint activity.

Building on our checkpoint mechanism, we have implemented two powerful facilities for experiment execution control: the ability to preemptively swap-out experiments without losing their run-time state, and the ability to time-travel through the run of a system.

REFERENCES

• 1: Paul Barham , Boris Dragovic , Keir Fraser , Steven Hand , Tim Harris , Alex Ho , Rolf Neugebauer , Ian Pratt , Andrew Warfield, Xen and the art of virtualization, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 2: K. Mani Chandy , Leslie Lamport, Distributed snapshots: determining global states of distributed systems, ACM Transactions on Computer Systems (TOCS), v.3 n.1, p.63-75, Feb. 1985 [doi>10.1145/214451.214456]
• 3: Ludmila Cherkasova , Rob Gardner, Measuring CPU overhead for I/O processing in the Xen virtual machine monitor, Proceedings of the annual conference on USENIX Annual Technical Conference, p.24-24, April 10-15, 2005, Anaheim, CA
• 4: Christopher Clark , Keir Fraser , Steven Hand , Jacob Gorm Hansen , Eric Jul , Christian Limpach , Ian Pratt , Andrew Warfield, Live migration of virtual machines, Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation, p.273-286, May 02-04, 2005
• 5: Russell Coker. Bonnie++, 2003. http://sourceforge.net/projects/bonnie/.
• 6: Brendan Cully , Geoffrey Lefebvre , Dutch Meyer , Mike Feeley , Norm Hutchinson , Andrew Warfield, Remus: high availability via asynchronous virtual machine replication, Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation, p.161-174, April 16-18, 2008, San Francisco, California
• 7: George W. Dunlap , Samuel T. King , Sukru Cinar , Murtaza A. Basrai , Peter M. Chen, ReVirt: enabling intrusion analysis through virtual-machine logging and replay, Proceedings of the 5th symposium on Operating systems design and implementation

  Due to copyright restrictions we are not able to make the PDFs for this conference available for downloading

  , December 09-11, 2002, Boston, Massachusetts [doi>10.1145/1060289.1060309]
• 8: George W. Dunlap , Dominic G. Lucchetti , Michael A. Fetterman , Peter M. Chen, Execution replay of multiprocessor virtual machines, Proceedings of the fourth ACM SIGPLAN/SIGOPS international conference on Virtual execution environments, March 05-07, 2008, Seattle, WA, USA [doi>10.1145/1346256.1346273]
• 9: E. N. (Mootaz) Elnozahy , Lorenzo Alvisi , Yi-Min Wang , David B. Johnson, A survey of rollback-recovery protocols in message-passing systems, ACM Computing Surveys (CSUR), v.34 n.3, p.375-408, September 2002 [doi>10.1145/568522.568525]
• 10: Dennis Geels et al. Friday: Global comprehension for distributed replay. In Proc. NSDI, pages 285--298, Cambridge, MA, April 2007.
• 11: Diwaker Gupta , Kenneth Yocum , Marvin McNett , Alex C. Snoeren , Amin Vahdat , Geoffrey M. Voelker, To infinity and beyond: time-warped network emulation, Proceedings of the 3rd conference on Networked Systems Design & Implementation, p.7-7, May 08-10, 2006, San Jose, CA
• 12: Diwaker Gupta , Kashi V. Vishwanath , Amin Vahdat, DieCast: testing distributed systems with an accurate scale model, Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation, p.407-422, April 16-18, 2008, San Francisco, California
• 13: Mike Hibler, Leigh Stoller, Jay Lepreau, Robert Ricci, and Chad Barb. Fast, scalable disk imaging with Frisbee. In Proc. USENIX, pages 283--296, San Antonio, TX, June 2003.
• 14: IEEE. IEEE 1558 standard for a precision clock synchronization protocol for networked measurement and control systems, September 2004.
• 15: Charles Killian et al. Life, death, and the critical transition: Finding liveness bugs in systems code. In Proc. NSDI, pages 243--256, Cambridge, MA, April 2007.
• 16: Samuel T. King , George W. Dunlap , Peter M. Chen, Debugging operating systems with time-traveling virtual machines, Proceedings of the annual conference on USENIX Annual Technical Conference, p.1-1, April 10-15, 2005, Anaheim, CA
• 17: H. Andres Lagar-Cavilla , Niraj Tolia , M. Satyanarayanan , Eyal de Lara, VMM-independent graphics acceleration, Proceedings of the 3rd international conference on Virtual execution environments, June 13-15, 2007, San Diego, California, USA [doi>10.1145/1254810.1254816]
• 18: Dutch T. Meyer , Gitika Aggarwal , Brendan Cully , Geoffrey Lefebvre , Michael J. Feeley , Norman C. Hutchinson , Andrew Warfield, Parallax: virtual disks for virtual machines, Proceedings of the 3rd ACM SIGOPS/EuroSys European Conference on Computer Systems 2008, April 01-04, 2008, Glasgow, Scotland UK
• 19: Jörg Micheel , Stephen Donnelly , Ian Graham, Precision timestamping of network packets, Proceedings of the 1st ACM SIGCOMM Workshop on Internet Measurement, November 01-02, 2001, San Francisco, California, USA [doi>10.1145/505202.505236]
• 20: David L. Mills. Internet time synchronization: The network time protocol. IEEE Trans. Comm., 39:1482--1493, 1991.
• 21: Steven Osman , Dinesh Subhraveti , Gong Su , Jason Nieh, The design and implementation of Zap: a system for migrating computing environments, Proceedings of the 5th symposium on Operating systems design and implementation

  Due to copyright restrictions we are not able to make the PDFs for this conference available for downloading

  , December 09-11, 2002, Boston, Massachusetts [doi>10.1145/1060289.1060323]
• 22: Feng Qin , Joseph Tucek , Jagadeesan Sundaresan , Yuanyuan Zhou, Rx: treating bugs as allergies---a safe method to survive software failures, Proceedings of the twentieth ACM symposium on Operating systems principles, October 23-26, 2005, Brighton, United Kingdom
• 23: Prashanth Radhakrishnan. Stateful-swapping in the Emulab network testbed. Master's thesis, University of Utah, August 2008.
• 24: Redhat. LVM2 Resource Page, 2006. http://sourceware.org/lvm2/.
• 25: Robert Ricci , Chris Alfeld , Jay Lepreau, A solver for the network testbed mapping problem, ACM SIGCOMM Computer Communication Review, v.33 n.2, April 2003 [doi>10.1145/956981.956988]
• 26: Robert Ricci et al. The Flexlab approach to realistic evaluation of networked systems. In Proc. NSDI, pages 201--214, Cambridge, MA, April 2007.
• 27: Luigi Rizzo, Dummynet: a simple approach to the evaluation of network protocols, ACM SIGCOMM Computer Communication Review, v.27 n.1, p.31-41, Jan. 1997 [doi>10.1145/251007.251012]
• 28: Jose Renato Santos , Yoshio Turner , G. Janakiraman , Ian Pratt, Bridging the gap between software and hardware techniques for I/O virtualization, USENIX 2008 Annual Technical Conference on Annual Technical Conference, p.29-42, June 22-27, 2008, Boston, Massachusetts
• 29: Sudarshan M. Srinivasan , Srikanth Kandula , Christopher R. Andrews , Yuanyuan Zhou, Flashback: a lightweight extension for rollback and deterministic replay for software debugging, Proceedings of the annual conference on USENIX Annual Technical Conference, p.3-3, June 27-July 02, 2004, Boston, MA
• 30: Sun Microsystems, Inc. ZFS, June 2008. http://www.opensolaris.org/os/community/zfs/.
• 31: Michael M. Swift , Muthukaruppan Annamalai , Brian N. Bershad , Henry M. Levy, Recovering device drivers, Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation, p.1-1, December 06-08, 2004, San Francisco, CA
• 32: Joseph Tucek , Shan Lu , Chengdu Huang , Spiros Xanthos , Yuanyuan Zhou, Triage: diagnosing production run failures at the user's site, Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles, October 14-17, 2007, Stevenson, Washington, USA
• 33: Darryl Veitch , Satish Babu , Attila Pàsztor, Robust synchronization of software clocks across the internet, Proceedings of the 4th ACM SIGCOMM conference on Internet measurement, October 25-27, 2004, Taormina, Sicily, Italy [doi>10.1145/1028788.1028817]
• 34: Brian White , Jay Lepreau , Leigh Stoller , Robert Ricci , Shashi Guruprasad , Mac Newbold , Mike Hibler , Chad Barb , Abhijeet Joglekar, An integrated experimental environment for distributed systems and networks, Proceedings of the 5th symposium on Operating systems design and implementation

  Due to copyright restrictions we are not able to make the PDFs for this conference available for downloading

  , December 09-11, 2002, Boston, Massachusetts [doi>10.1145/1060289.1060313]
• 35: Junfeng Yang , Paul Twohey , Dawson Engler , Madanlal Musuvathi, Using model checking to find serious file system errors, ACM Transactions on Computer Systems (TOCS), v.24 n.4, p.393-423, November 2006 [doi>10.1145/1189256.1189259]
• 36: Andreas Zeller, Isolating cause-effect chains from computer programs, Proceedings of the 10th ACM SIGSOFT symposium on Foundations of software engineering, November 18-22, 2002, Charleston, South Carolina, USA [doi>10.1145/587051.587053]

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.5 Reliability
          Subjects: Checkpoint/restart

Additional Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.7 Organization and Design
          Subjects: Distributed systems

General Terms:
Design, Performance

Keywords:
distributed checkpointing, emulab, network testbed, transparent checkpointing