Transparent logging as a technique for debugging complex distributed systems

ABSTRACT

As any battle-scarred veteran will testify, debugging a distributed system in production use is an enterprise fraught with great difficulty and frustration. By the time the system is released for production use, most of the easy bugs have been found and fixed. The remaining bugs are typically non-deterministic in nature, and will only manifest themselves under conditions of heavy use. Although rare, such bugs cannot be ignored because they often have serious consequences.In this position paper, we put forth the thesis that logging is a flexible, powerful, and convenient tool for debugging complex distributed systems. We substantiate this thesis in three steps. First, we argue that logging is particularly well suited for debugging distributed systems. Next, we observe that logging is already used in distributed systems for reasons independent of debugging. Finally, we show that the latter uses of logging can be transparently extended to support debugging.

REFERENCES

• 1: J. Gray. Notes on data base operating systems. Technical Report RJ2188, IBM Research Laboratory, San Jose, February 1978.
• 2: R. Hagmann, Reimplementing the Cedar file system using logging and group commit, Proceedings of the eleventh ACM Symposium on Operating systems principles, p.155-162, November 08-11, 1987, Austin, Texas, United States
• 3: H. Mashburn and M. Satyanarayanan. RVM: Recoverable Virtual Memory User Manual. School of Computer Science, Carnegie Mellon University, April 1991.
• 4: Mendel Rosenblum , John K. Ousterhout, The design and implementation of a log-structured file system, ACM Transactions on Computer Systems (TOCS), v.10 n.1, p.26-52, Feb. 1992 [doi>10.1145/146941.146943]
• 5: M. Satyanarayanan, Integrating security in a large distributed system, ACM Transactions on Computer Systems (TOCS), v.7 n.3, p.247-280, Aug. 1989 [doi>10.1145/65000.65002]
• 6: Mahadev Satyanarayanan, Scalable, Secure, and Highly Available Distributed File Access, Computer, v.23 n.5, p.9-18, 20-21, May 1990 [doi>10.1109/2.53351]
• 7: M. Satyanarayanan, J. Kistler, P. Kumar, and M. Mashburn. On the ubiquity of logging in distributed file systems. In Proceedings of the Third Workshop on Workstation Operating Systems, April 1992.