Fastpath Optimizations for Cluster Recovery in Shared-Disk Systems

ABSTRACT

We describe the design and implementation of a clustering service for a high-performance, shared-disk file system. The service provides failure detection and recovery, reliableend-to-end messaging, and a centralized and recoverable management interface. We implement novel optimizations in the voting protocol that resolves cluster membership. Optimizations allow clusters to form as quickly as possible without introducing livelock or requiring timeout parameters to be tuned carefully. Our treatment includes performance results that quantify the scalability of the system and measure recovery times.

REFERENCES

• 1: Yair Amir , Giuseppe Ateniese , Damian Hasse , Yongdae Kim , Cristina Nita-Rotaru , Theo Schlossnagle , John Schultz , Jonathan Stanton , Gene Tsudik, Secure Group Communication in Asynchronous Networks with Failures: Integration and Experiments, Proceedings of the The 20th International Conference on Distributed Computing Systems ( ICDCS 2000), p.330, April 10-13, 2000
• 2: [2] Y. Amir and J. Stanton. The spread wide area group communication system. Technical Report CNDS-98-4, Center for Network and Distributed Systems, Johns Hopkins University, 1998.
• 3: Mary Louise Gray Baker, Fast crash recovery in distributed file systems, University of California at Berkeley, Berkeley, CA, 1995
• 4: Kenneth P. Birman , Robert V. Renesse , Robbert Van Renesse, Reliable Distributed Computing with the ISIS Toolkit, IEEE Computer Society Press, Los Alamitos, CA, 1994
• 5: [5] Zoning implementation strategies for brocade SAN fabrics. Brocade Inc., White Paper, 2002.
• 6: Randal Chilton Burns , Darrell D. Long, Data management in a distributed file system for storage area networks, University of California, Santa Cruz, 2000
• 7: [7] R. Burns, R. M. Rees, and D. D. E. Long. Safe caching in a distributed file system. In Proceedings of the International Parallel and Distributed Processing Symposium, 2000.
• 8: [8] D. Naor et al. Object store security document. Storage Networking Industry Association (SNIA), 2003.
• 9: Murthy Devarakonda , Bill Kish , Ajay Mohindra, Recovery in the Calypso file system, ACM Transactions on Computer Systems (TOCS), v.14 n.3, p.287-310, Aug. 1996 [doi>10.1145/233557.233560]
• 10: A. El Abbadi , S. Toueg, Maintaining availability in partitioned replicated databases, ACM Transactions on Database Systems (TODS), v.14 n.2, p.264-290, June 1989 [doi>10.1145/63500.63501]
• 11: Eli Gafni , Leslie Lamport, Disk Paxos, Proceedings of the 14th International Conference on Distributed Computing, p.330-344, October 04-06, 2000
• 12: Howard Bradley Gobioff , Garth Gibson , Doug Tygar, Security for a high performance commodity storage subsystem, Carnegie Mellon University, Pittsburgh, PA, 1999
• 13: Lifan Gu , J. J. Garcia-Luna-Aceves, New Error Recovery Structures for Reliable Multicasting, Proceedings of the 6th International Conference on Computer Communications and Networks, p.189, September 22-25, 1997
• 14: Katherine Guo , Werner Vogels , Robbert van Renesse, Structured virtual synchrony: exploring the bounds of virtual synchronous group communication, Proceedings of the 7th workshop on ACM SIGOPS European workshop: Systems support for worldwide applications, September 09-11, 1996, Connemara, Ireland [doi>10.1145/504450.504488]
• 15: [15] RS/6000 SP high availability infrastructure. IBM Redbook SG224-4838, IBM, 1996.
• 16: Availability Study of Dynamic Voting Algorithms, Proceedings of the The 21st International Conference on Distributed Computing Systems, p.247, April 16-19, 2001
• 17: [17] F. Jahanian, R. Rajkumar, and S. Fakhouri. Processor group membership protocols: Specification, design, and implementation. In Proceedings of the IEEE Symposium on Reliable Distributed Systems, 1993.
• 18: S. Jajodia , David Mutchler, Dynamic voting algorithms for maintaining the consistency of a replicated database, ACM Transactions on Database Systems (TODS), v.15 n.2, p.230-280, June 1990 [doi>10.1145/78922.78926]
• 19: Idit Keidar , Danny Dolev, Increasing the resilience of atomic commit, at no additional cost, Proceedings of the fourteenth ACM SIGACT-SIGMOD-SIGART symposium on Principles of database systems, p.245-254, May 22-25, 1995, San Jose, California, United States [doi>10.1145/212433.212468]
• 20: Leslie Lamport, The part-time parliament, ACM Transactions on Computer Systems (TOCS), v.16 n.2, p.133-169, May 1998 [doi>10.1145/279227.279229]
• 21: Jehan-François Pâris , Darrell D. E. Long, Efficient Dynamic Voting Algorithms, Proceedings of the Fourth International Conference on Data Engineering, p.268-275, February 01-05, 1988
• 22: Nancy A. Lynch , Alexander A. Shvartsman, RAMBO: A Reconfigurable Atomic Memory Service for Dynamic Networks, Proceedings of the 16th International Conference on Distributed Computing, p.173-190, October 28-30, 2002
• 23: Danny Dolev , Dalia Malki, The Transis approach to high availability cluster communication, Communications of the ACM, v.39 n.4, p.64-70, April 1996 [doi>10.1145/227210.227227]
• 24: [24] C. Malloth and K. Schiper. View synchronous communication in large scale networks. In Workshop of the ESPRIT project BROADCAST, number 6360, 1995.
• 25: J. Menon , D. A. Pease , R. Rees , L. Duyanovich , B. Hillsberg, IBM Storage Tank-- A heterogeneous scalable SAN file system, IBM Systems Journal, v.42 n.2, p.250-267, April 2003
• 26: C. Mohan , Don Haderle , Bruce Lindsay , Hamid Pirahesh , Peter Schwarz, ARIES: a transaction recovery method supporting fine-granularity locking and partial rollbacks using write-ahead logging, ACM Transactions on Database Systems (TODS), v.17 n.1, p.94-162, March 1992 [doi>10.1145/128765.128770]
• 27: [27] L. E. Moser, Y. Amir, P. M. Melliar-Smith, and D. A. Agarwal. Extended virtual synchrony. In The IEEE International Conference on Distributed Computing Systems (ICDCS), 1994.
• 28: Providing Support for Survivable CORBA Applications with the Immune System, Proceedings of the 19th IEEE International Conference on Distributed Computing Systems, p.507, May 31-June 04, 1999
• 29: [29] J. Palmer, R. Strong, and E. Upfal. Nonblocking ordered reliable multicast in an unreliable distributed environment. Technical Report RJ-10096 (91913), IBM Research Division, 1997.
• 30: Benjamin C. Reed , Edward G. Chron , Randal C. Burns , Darrell D. E. Long, Authenticating Network-Attached Storage, IEEE Micro, v.20 n.1, p.49-57, January 2000 [doi>10.1109/40.820053]
• 31: The architecture and performance of security protocols in the ensemble group communication system: Using diamonds to guard the castle, ACM Transactions on Information and System Security (TISSEC), v.4 n.3, p.289-319, August 2001 [doi>10.1145/501978.501982]
• 32: [32] L. Rodrigues and P. Verissimo. xAMp: A protocol suite for group communication. Technical Report RT/43-92, INSEC, 1992.
• 33: Frank B. Schmuck , Roger L. Haskin, GPFS: A Shared-Disk File System for Large Computing Clusters, Proceedings of the Conference on File and Storage Technologies, p.231-244, January 28-30, 2002
• 34: [34] D. Skeen. A quorum-based commit protocol. In Workshop of Distributed Data Management and Computer Networks, 1982.
• 35: Chandramohan A. Thekkath , Timothy Mann , Edward K. Lee, Frangipani: a scalable distributed file system, Proceedings of the sixteenth ACM symposium on Operating systems principles, p.224-237, October 05-08, 1997, Saint Malo, France