Quantifying and Improving the Availability of High-Performance Cluster-Based Internet Services

ABSTRACT

Cluster-based servers can substantially increase performance when nodes cooperate to globally manage resources. However, in this paper we show that cooperation results in a substantial availability loss, in the absence of high-availability mechanisms. Specifically, we show that a sophisticated cluster-based Web server, which gains a factor of 3 in performance through cooperation, increases service unavailability by a factor of 10 over a non-cooperative version. We then show how to augment this Web server with software components embodying a small set of high-availability techniques to regain the lost availability. Among other interesting observations, we show that the application of multiple high-availability techniques, each implemented independently in its own subsystem, can lead to inconsistent recovery actions. We also show that a novel technique called Fault Model Enforcement can be used to resolve such inconsistencies. Augmenting the server with these techniques led to a final expected availability of close to 99.99%.

REFERENCES

• 1: Satoshi Asami, Reducing the Cost of System Administration of a Disk Storage System, University of California at Berkeley, Berkeley, CA, 2000
• 2: Saurabh Bagchi , Balaji Srinivasan , Keith Whisnant , Zbigniew Kalbarczyk , Ravishankar K. Iyer, Hierarchical Error Detection in a Software Implemented Fault Tolerance (SIFT) Environment, IEEE Transactions on Knowledge and Data Engineering, v.12 n.2, p.203-224, March 2000 [doi>10.1109/69.842263]
• 3: Ricardo Bianchini , Enrique V. Carrera, Analytical and experimental evaluation of cluster-based network servers, World Wide Web, v.3 n.4, p.215-229, December 2000 [doi>10.1023/A:1018734706376]
• 4: Eric A. Brewer, Lessons from Giant-Scale Services, IEEE Internet Computing, v.5 n.4, p.46-55, July 2001 [doi>10.1109/4236.939450]
• 5: Enrique V. Carrera , Ricardo Bianchini, Efficiency vs. portability in cluster-based network servers, Proceedings of the eighth ACM SIGPLAN symposium on Principles and practices of parallel programming, p.113-122, June 2001, Snowbird, Utah, United States
• 6: Srinath Rao, User-Level Communication in Cluster-Based Servers, Proceedings of the 8th International Symposium on High-Performance Computer Architecture, p.275, February 02-06, 2002
• 7: Mike Y. Chen , Emre Kiciman , Eugene Fratkin , Armando Fox , Eric Brewer, Pinpoint: Problem Determination in Large, Dynamic Internet Services, Proceedings of the 2002 International Conference on Dependable Systems and Networks, p.595-604, June 23-26, 2002
• 8: [8] Cisco CSS 11500 Series Content Services Switches, Apr. 2003. Available at http://www.cisco.com/en/US/products/hw/contnetw/ps792/ index.html.
• 9: Diamantino Costa , Tiago Rilho , Henrique Madeira, Joint Evaluation of Performance and Robustness of a COTS DBMS through Fault-Injection, Proceedings of the 2000 International Conference on Dependable Systems and Networks (formerly FTCS-30 and DCCA-8), p.251-260, June 25-28, 2000
• 10: [10] F. Cristian and F. Schmuck. Agreeing on Processor Group Membership in Timed Asynchronous Distributed Systems. 1995.
• 11: Armando Fox , Eric A. Brewer, Harvest, Yield, and Scalable Tolerant Systems, Proceedings of the The Seventh Workshop on Hot Topics in Operating Systems, p.174, March 28-30, 1999
• 12: [12] J. Gray. A Census of Tandem System Availability Between 1985 and 1990. IEEE Transactions on Reliability, 39(4):409- 418, Oct. 1990.
• 13: Steven D. Gribble , Eric A. Brewer , Joseph M. Hellerstein , David Culler, Scalable, distributed data structures for internet service construction, Proceedings of the 4th conference on Symposium on Operating System Design & Implementation, p.22-22, October 22-25, 2000, San Diego, California
• 14: Taliver Heath , Richard P. Martin , Thu D. Nguyen, Improving cluster availability using workstation validation, Proceedings of the 2002 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, June 15-19, 2002, Marina Del Rey, California
• 15: Zbigniew T. Kalbarczyk , Ravishankar K. Iyer , Saurabh Bagchi , Keith Whisnant, Chameleon: A Software Infrastructure for Adaptive Fault Tolerance, IEEE Transactions on Parallel and Distributed Systems, v.10 n.6, p.560-579, June 1999 [doi>10.1109/71.774907]
• 16: M. Kalyanakrishnam , Z. Kalbarczyk , R. Iyer, Failure Data Analysis of a LAN of Windows NT Based Computers, Proceedings of the 18th IEEE Symposium on Reliable Distributed Systems, p.178, October 18-21, 1999
• 17: [17] I. Lee and R. Iyer. Faults, Symptoms, and Software Fault Tolerance in the Tandem GUARDIAN90 Operating System. In Proceedings of International Symposium on Fault-Tolerant Computing (FTCS-23), pages 20-29, 1993.
• 18: Lei Li , Kalyanaraman Vaidyanathan , Kishor S. Trivedi, An Approach for Estimation of Software Aging in a Web Server, Proceedings of the 2002 International Symposium on Empirical Software Engineering, p.91, October 03-04, 2002
• 19: [19] X. Li, R. P. Martin, K. Nagaraja, T. D. Nguyen, and B. Zhang. Mendosus: A SAN-Based Fault-Injection Test-Bed for the Construction of Highly Available Network Services. In Proceedings of the 1st Workshop on Novel Uses of System Area Networks (SAN-1), Cambridge, MA, Jan. 2002.
• 20: [20] Linux virtual server project. http://www.linuxvirtualserver.org/.
• 21: [21] D. D. E. Long, J. L. Carroll, and C. J. Park. A Study of the Reliability of Internet Sites. In Proceedings of the Tenth Symposium on Reliable Distributed Systems, pages 177-186, Sept. 1991.
• 22: [22] B. Murphy and B. Levidow. Windows 2000 Dependability. (MSR-TR-2000-56), June 2000.
• 23: [23] K. Nagaraja, R. Bianchini, R. Martin, and T. D. Nguyen. Using Fault Model Enforcement to Improve Availability. In Proceedings of the Second Workshop on Evaluating and Architecting System dependabilitY (EASY), Oct. 2002.
• 24: Kiran Nagaraja , Xiaoyan Li , Ricardo Bianchini , Richard P. Martin , Thu D. Nguyen, Using fault injection and modeling to evaluate the performability of cluster-based services, Proceedings of the 4th conference on USENIX Symposium on Internet Technologies and Systems, p.2-2, March 26-28, 2003, Seattle, WA
• 25: Vivek S. Pai , Mohit Aron , Gaurov Banga , Michael Svendsen , Peter Druschel , Willy Zwaenepoel , Erich Nahum, Locality-aware request distribution in cluster-based network servers, Proceedings of the eighth international conference on Architectural support for programming languages and operating systems, p.205-216, October 02-07, 1998, San Jose, California, United States
• 26: David A. Patterson , Garth Gibson , Randy H. Katz, A case for redundant arrays of inexpensive disks (RAID), Proceedings of the 1988 ACM SIGMOD international conference on Management of data, p.109-116, June 01-03, 1988, Chicago, Illinois, United States
• 27: Yasushi Saito , Brian N. Bershad , Henry M. Levy, Manageability, availability and performance in Porcupine: a highly scalable, cluster-based mail service, Proceedings of the seventeenth ACM symposium on Operating systems principles, p.1-15, December 12-15, 1999, Charleston, South Carolina, United States
• 28: [28] Service Monitoring Daemon, Apr. 2003. Available at http://www.kernel.org/software/mon/.
• 29: [29] M. Sullivan and R. Chillarege. Software Defects and their Impact on System Availability - A Study of Field Failures in Operating Systems. In Proceedings of the 21st International Symposium on Fault-Tolerant Computing (FTCS-21), pages 2-9, Montreal, Canada, 1991.
• 30: T. K. Tsai , R. K. Iyer , D. Jewitt, An approach towards benchmarking of fault-tolerant commercial systems, Proceedings of the The Twenty-Sixth Annual International Symposium on Fault-Tolerant Computing (FTCS '96), p.314, June 25-27, 1996
• 31: Matt Welsh , David Culler , Eric Brewer, SEDA: an architecture for well-conditioned, scalable internet services, Proceedings of the eighteenth ACM symposium on Operating systems principles, October 21-24, 2001, Banff, Alberta, Canada