On application-level approaches to avoiding TCP throughput collapse in cluster-based storage systems

ABSTRACT

TCP Incast plagues scalable cluster-based storage built atop standard TCP/IP-over-Ethernet, often resulting in much lower client read bandwidth than can be provided by the available network links. This paper reviews the Incast problem and discusses potential application-level approaches to avoiding it.

REFERENCES

• 1: Michael Abd-El-Malek , William V. Courtright, II , Chuck Cranor , Gregory R. Ganger , James Hendricks , Andrew J. Klosterman , Michael Mesnier , Manish Prasad , Brandon Salmon , Raja R. Sambasivan , Shafeeq Sinnamohideen , John D. Strunk , Eno Thereska , Matthew Wachs , Jay J. Wylie, Ursa minor: versatile cluster-based storage, Proceedings of the 4th conference on USENIX Conference on File and Storage Technologies, p.5-5, December 13-16, 2005, San Francisco, CA
• 2: M. Allman , H. Balakrishnan , S. Floyd, Enhancing TCP's Loss Recovery Using Limited Transmit, RFC Editor, 2001
• 3: Mark Allman , Vern Paxson, On estimating end-to-end network path properties, ACM SIGCOMM Computer Communication Review, v.31 n.2 supplement, April 2001 [doi>10.1145/844193.844203]
• 4: M. Allman , V. Paxson , W. Stevens, TCP Congestion Control, RFC Editor, 1999
• 5: Remzi H. Arpaci-Dusseau , Andrea C. Arpaci-Dusseau, Fail-Stutter Fault Tolerance, Proceedings of the Eighth Workshop on Hot Topics in Operating Systems, p.33, May 20-22, 2001
• 6: P. J. Braam. File systems for clusters from a protocol perspective. In Second Extreme Linux Topics Workshop, June 1999.
• 7: J. Butler. Panasas Inc. Personal Communication, August 2007.
• 8: S. Floyd , T. Henderson , A. Gurtov, The NewReno Modification to TCP's Fast Recovery Algorithm, RFC Editor, 2004
• 9: Sanjay Ghemawat , Howard Gobioff , Shun-Tak Leung, The Google file system, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 10: Garth A. Gibson , David F. Nagle , Khalil Amiri , Jeff Butler , Fay W. Chang , Howard Gobioff , Charles Hardin , Erik Riedel , David Rochberg , Jim Zelenka, A cost-effective, high-bandwidth storage architecture, Proceedings of the eighth international conference on Architectural support for programming languages and operating systems, p.92-103, October 02-07, 1998, San Jose, California, United States
• 11: G. Grider, H. Chen, J. Junes., S. Poole, R. Wacha, P. Fields, R. Martinez, S. Khalsa, A. Matthews, and G. Gibson. PaScal - A New Parallel and Scalable Server IO Networking Infrastructure for Supporting Global Storage/File Systems in Large-size Linux Clusters. In Proceedings of the 25th IEEE International Performance Computing and Communications Conference, Phoenix, AZ, Apr. 2006.
• 12: Charles E. Leiserson, Fat-trees: universal networks for hardware-efficient supercomputing, IEEE Transactions on Computers, v.34 n.10, p.892-901, Oct. 1985
• 13: David Nagle , Denis Serenyi , Abbie Matthews, The Panasas ActiveScale Storage Cluster: Delivering Scalable High Bandwidth Storage, Proceedings of the 2004 ACM/IEEE conference on Supercomputing, p.53, November 06-12, 2004 [doi>10.1109/SC.2004.57]
• 14: A. Phanishayee, E. Krevat, V. Vasudevan, D. Andersen, G. Ganger, G. Gibson, and S. Seshan. Measurement and Analysis of TCP Throughput Collapse in Cluster-based Storage Systems. Technical Report CMU-PDL-07-105, Sept 2007.
• 15: Frank B. Schmuck , Roger L. Haskin, GPFS: A Shared-Disk File System for Large Computing Clusters, Proceedings of the Conference on File and Storage Technologies, p.231-244, January 28-30, 2002
• 16: M. Wadekar. Enhanced ethernet for data center: Reliable, channelized and robust. In 15th IEEE Workshop on Local and Metropolitan Area Networks, June 2007.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.4 Distributed Systems

Additional Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.5 Local and Wide-Area Networks
  C.4 PERFORMANCE OF SYSTEMS

General Terms:
Design, Performance

Keywords:
Incast, TCP, cluster-based storage