Practical and low-overhead masking of failures of TCP-based servers

ABSTRACT

This article describes an architecture that allows a replicated service to survive crashes without breaking its TCP connections. Our approach does not require modifications to the TCP protocol, to the operating system on the server, or to any of the software running on the clients. Furthermore, it runs on commodity hardware. We compare two implementations of this architecture (one based on primary/backup replication and another based on message logging) focusing on scalability, failover time, and application transparency. We evaluate three types of services: a file server, a Web server, and a multimedia streaming server. Our experiments suggest that the approach incurs low overhead on throughput, scales well as the number of clients increases, and allows recovery of the service in near-optimal time.

REFERENCES

• 1: Aghdaie, N. and Tamir, Y. 2002. Implementation and evaluation of transparent fault-tolerant web service with kernel-level support. In Proceedings of the 11th IEEE International Conference on Computer Communications and Networks (ICCCN), 63--68.
• 2: Aghdaie, N. and Tamir, Y. 2003. Fast transparent failover for reliable web service. In Proceedings of the 15th IASTED International Conference on Parallel and Distributed Computing and Systems (PDCS).
• 3: Alvisi, L., Bressoud, T. C., El-Khashab, A., Marzullo, K., and Zagorodnov, D. 2001. Wrapping server-side TCP to mask connection failures. In Proceedings of the IEEE InfoCom Conference, 329--337.
• 4: Apache. 2005. Apache homepage. http://www.apache.org/.
• 5: Basile, C., Kalbarczyk, Z., and K., I. R. 2003. A preemptive deterministic scheduling algorithm for multithreaded replicas. In Proceedings of the International Conference on Dependable Systems and Networks (DSN), 149--158.
• 6: Basile, C., Kalbarczyk, Z., Whisnant, K., and Iyer, R. K. 2002. Active replication of multithreaded applications. Tech. rep. CRHC-02-01, University of Illinois.
• 7: Bhide, A., Elnozahy, E., and Morgan, S. 1991. A highly available network file server. In Proceedings of the USENIX Winter Technical Conference, 199--205.
• 8: Robert Bradford , Evangelos Kotsovinos , Anja Feldmann , Harald Schiöberg, Live wide-area migration of virtual machines including local persistent state, Proceedings of the 3rd international conference on Virtual execution environments, June 13-15, 2007, San Diego, California, USA [doi>10.1145/1254810.1254834]
• 9: T. C. Bressoud, TFT: A Software System for Application-Transparent Fault Tolerance, Proceedings of the The Twenty-Eighth Annual International Symposium on Fault-Tolerant Computing, p.128, June 23-25, 1998
• 10: Thomas C. Bressoud , Fred B. Schneider, Hypervisor-based fault tolerance, ACM Transactions on Computer Systems (TOCS), v.14 n.1, p.80-107, Feb. 1996 [doi>10.1145/225535.225538]
• 11: Budhiraja, N., Marzullo, K., Schneider, F., and Toueg, S. 1992. Primary-Backup protocols: Lower bounds and optimal implementations. In Proceedings of the 3rd IFIP Conference on Dependable Computing for Critical Applications, 187--198.
• 12: HotSwap-Transparent Server Failover for Linux, Proceedings of the 16th USENIX conference on System administration, November 03-08, 2002, Philadelphia, PA
• 13: Christopher Clark , Keir Fraser , Steven Hand , Jacob Gorm Hansen , Eric Jul , Christian Limpach , Ian Pratt , Andrew Warfield, Live migration of virtual machines, Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation, p.273-286, May 02-04, 2005
• 14: Brendan Cully , Geoffrey Lefebvre , Dutch Meyer , Mike Feeley , Norm Hutchinson , Andrew Warfield, Remus: high availability via asynchronous virtual machine replication, Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation, p.161-174, April 16-18, 2008, San Francisco, California
• 15: Daniel, E. and Choi, G. S. 1999. TMR for off-the-shelf Unix systems. Short presentation at IEEE International Symposium on Fault-Tolerant Computing (FTCS).
• 16: Dolev, D., Malki, D., and Yarom, Y. 1994. Warm backup using snooping. In Proceedings of the 1st International Workshop on Services in Distributed and Networked Environments (SDNE), 60--65.
• 17: DSS. 2005. Homepage. http://developer.apple.com/darwin/projects/streaming/.
• 18: Robust TCP Connections for Fault Tolerant Computing, Proceedings of the 9th International Conference on Parallel and Distributed Systems, p.501, December 17-20, 2002
• 19: E. N. (Mootaz) Elnozahy , Lorenzo Alvisi , Yi-Min Wang , David B. Johnson, A survey of rollback-recovery protocols in message-passing systems, ACM Computing Surveys (CSUR), v.34 n.3, p.375-408, September 2002 [doi>10.1145/568522.568525]
• 20: Fetzer, C. and Mishra, S. 1999. Transparent TCP/IP based replication. Short presentation at IEEE International Symposium on Fault-Tolerant Computing (FTCS).
• 21: Christopher Hertel, Implementing CIFS: The Common Internet File System, Prentice Hall Professional Technical Reference, 2003
• 22: V. Jacobson, Congestion avoidance and control, ACM SIGCOMM Computer Communication Review, v.18 n.4, p.314-329, August 1988
• 23: Koch, R. R., Hortikar, S., E., M. L., and M., M.-S. P. 2003. Transparent TCP connection failover. In Proceedings of the IEEE International Conference on Dependable Systems and Networks (DSN), 383--392.
• 24: Luo, M. and Yang, C. 2001. Constructing zero-loss web services. In Proceedings of the IEEE InfoCom, 1781--1790.
• 25: Marwah, M., Mishra, S., and Fetzer, C. 2003. TCP server fault tolerance using connection migration to a backup server. In Proceedings of the IEEE International Conference on Dependable Systems and Networks (DSN), 373--382.
• 26: Manish Marwah , Christof Fetzer, A System Demonstration of ST-TCP, Proceedings of the 2005 International Conference on Dependable Systems and Networks, p.308-313, June 28-July 01, 2005 [doi>10.1109/DSN.2005.14]
• 27: J. Nagle, Congestion control in IP/TCP internetworks, RFC Editor, 1984
• 28: Napper, J., Alvisi, L., and Vin, H. 2003. A fault-tolerant Java virtual machine. In Proceedings of the IEEE International Conference on Dependable Systems and Networks (DSN), 425--434.
• 29: Nasika, R. and Dasgupta, P. 2000. Transparent migration of distributed communicating processes. In Proceedings of the 13th ISCA International Conference on Parallel and Distributed Computing Systems (PDCS).
• 30: Maxim Orgiyan , Christof Fetzer, Tapping TCP Streams, Proceedings of the IEEE International Symposium on Network Computing and Applications (NCA'01), p.278, October 08-10, 2001
• 31: V. Paxson , M. Allman, Computing TCP's Retransmission Timer, RFC Editor, 2000
• 32: N. Peyrouze , G. Muller, FT-NFS: an efficient fault-tolerant NFS server designed for off-the-shelf workstations, Proceedings of the The Twenty-Sixth Annual International Symposium on Fault-Tolerant Computing (FTCS '96), p.64, June 25-27, 1996
• 33: Michael L. Powell , David L. Presotto, Publishing: a reliable broadcast communication mechanism, Proceedings of the ninth ACM symposium on Operating systems principles, p.100-109, October 10-13, 1983, Bretton Woods, New Hampshire, United States
• 34: Eric Rescorla , Adam Cain , Brian Korver, SSLACC: A Clustered SSL Accelerator, Proceedings of the 11th USENIX Security Symposium, p.229-246, August 05-09, 2002
• 35: A. Rijsinghani, Computation of the Internet Checksum via Incremental Update, RFC Editor, 1994
• 36: Rodrigo Rodrigues , Miguel Castro , Barbara Liskov, BASE: using abstraction to improve fault tolerance, Proceedings of the eighteenth ACM symposium on Operating systems principles, October 21-24, 2001, Banff, Alberta, Canada
• 37: Pasi Sarolahti , Alexey Kuznetsov, Congestion Control in Linux TCP, Proceedings of the FREENIX Track: 2002 USENIX Annual Technical Conference, p.49-62, June 10-15, 2002
• 38: Gurudatt Shenoy , Suresh K. Satapati , Riccardo Bettati, HYDRANET-FT: Network Support for Dependable Services, Proceedings of the The 20th International Conference on Distributed Computing Systems ( ICDCS 2000), p.699, April 10-13, 2000
• 39: Joseph G. Slember , Priya Narasimhan, Using Program Analysis to Identify and Compensate for Nondeterminism in Fault-Tolerant, Replicated Systems, Proceedings of the 23rd IEEE International Symposium on Reliable Distributed Systems, p.251-263, October 18-20, 2004
• 40: J. H. Slye , E. N. Elnozahy, Supporting nondeterministic execution in fault-tolerant systems, Proceedings of the The Twenty-Sixth Annual International Symposium on Fault-Tolerant Computing (FTCS '96), p.250, June 25-27, 1996
• 41: SMB. 2005. Samba homepage. http://www.samba.org/.
• 42: Alex C. Snoeren , David G. Andersen , Hari Balakrishnan, Fine-grained failover using connection migration, Proceedings of the 3rd conference on USENIX Symposium on Internet Technologies and Systems, p.19-19, March 26-28, 2001, San Francisco, California
• 43: Srinivasan, K. 2001. M-TCP: Transport layer support for highly available network services. M.S. thesis, Rutgers University. Available as Tech. Rep. DCS-TR-459.
• 44: P. Srisuresh , M. Holdrege, IP Network Address Translator (NAT) Terminology and Considerations, RFC Editor, 1999
• 45: W. Richard Stevens, TCP/IP illustrated (vol. 1): the protocols, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1993
• 46: Florin Sultan , Aniruddha Bohra , Stephen Smaldone , Yufei Pan , Pascal Gallard , Iulian Neamtiu , Liviu Iftode, Recovering Internet Service Sessions from Operating System Failures, IEEE Internet Computing, v.9 n.2, p.17-27, March 2005 [doi>10.1109/MIC.2005.45]
• 47: Sultan, F., Bohra, A., and Iftode, L. 2003. Service continuations: An operating system mechanism for dynamic migration of Internet service sessions. In Proceedings of the Symposium Reliable Distributed Systems (SRDS), 177--186.
• 48: Florin Sultan , Kiran Srinivasan , Deepa Iyer , Liviu Iftode, Migratory TCP: Connection Migration for Service Continuity in the Internet, Proceedings of the 22 nd International Conference on Distributed Computing Systems (ICDCS'02), p.469, July 02-05, 2002
• 49: Sultan, F., Srinivasan, K., and Iftode, L. 2001. Transport layer support for highly-available network services. Tech. rep. DCS-TR-429, Rutgers University, May.
• 50: X/Open. 1992. Protocols for X/Open PC Interworking: SMB, Version 2. X/Open Company Ltd. Also available at http://www.opengroup.org/products/publications/catalog/c209.htm.
• 51: Chu-Sing Yang , Mon-Yen Luo, Realizing fault resilience in Web-server cluster, Proceedings of the 2000 ACM/IEEE conference on Supercomputing (CDROM), p.21-es, November 04-10, 2000, Dallas, Texas, United States
• 52: Zagorodnov, D. and Marzullo, K. 2005. Managing self-inflicted nondeterminism. In Proceedings of the 1st Workshop on Hot Topics in System Dependability (HotDep), 323--328.
• 53: Zagorodnov, D., Marzullo, K., Alvisi, L., and Bressoud, T. 2003. Engineering fault-tolerant TCP/IP servers using FT-TCP. In Proceedings of the IEEE International Conference on Dependable Systems and Networks (DSN), 393--402.
• 54: Victor C. Zandy , Barton P. Miller, Reliable network connections, Proceedings of the 8th annual international conference on Mobile computing and networking, September 23-28, 2002, Atlanta, Georgia, USA [doi>10.1145/570645.570657]
• 55: Zhang, R., Abdelzaher, T. F., and Stankovic, J. A. 2004. Efficient TCP connection failover in web server clusters. In Proceedings of the IEEE InfoCom Conference. Vol. 2, 1219--1228.

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.4 Communications Management
          Subjects: Network communication

Additional Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.4 Distributed Systems
          Subjects: Network operating systems
      C.2.5 Local and Wide-Area Networks
          Subjects: Internet (e.g., TCP/IP)

  D. Software
  D.4 OPERATING SYSTEMS
      D.4.5 Reliability
          Subjects: Fault-tolerance
      D.4.8 Performance
          Subjects: Measurements

General Terms:
Algorithms, Performance, Reliability

Keywords:
Fault-tolerant computing system, TCP/IP, primary/backup approach