Experimental study of router buffer sizing

ABSTRACT

During the past four years, several papers have proposed rules for sizing buffers in Internet core routers. Appenzeller et al. suggest that a link needs a buffer of size O(C/√N), where C is the capacity of the link, and N is the number of flows sharing the link. If correct, buffers could be reduced by 99% in a typical backbone router today without loss in throughput. Enachecsu et al., and Raina et al. suggest that buffers can be reduced even further to 20-50 packets if we are willing to sacrifice a fraction of link capacities, and if there is a large ratio between the speed of core and access links. If correct, this is a five orders of magnitude reduction in buffer sizes. Each proposal is based on theoretical analysis and validated using simulations. Given the potential benefits (and the risk of getting it wrong!) it is worth asking if these results hold in real operational networks. In this paper, we report buffer-sizing experiments performed on real networks - either laboratory networks with commercial routers as well as customized switching and monitoring equipment (UW Madison, Sprint ATL, and University of Toronto), or operational backbone networks (Level 3 Communications backbone network, Internet2, and Stanford). The good news: Subject to the limited scenarios we can create, the buffer sizing results appear to hold. While we are confident that the O(C/√N) will hold quite generally for backbone routers, the 20-50 packet rule should be applied with extra caution to ensure that network components satisfy the underlying assumptions.

REFERENCES

• 1: NetFPGA project. http://yuba.stanford.edu/NetFPGA/.
• 2: Sprintlink ip backbone measurements. IPMON Sprint Labs IP Monitoring Project, http://ipmon.sprint.com.
• 3: A. Aggarwal, S. Savage, and T. Anderson. Understanding the performance of TCP pacing. In Proceedings of the IEEE INFOCOM, pages 1157--1165, Tel-Aviv, Israel, March 2000.
• 4: Guido Appenzeller , Isaac Keslassy , Nick McKeown, Sizing router buffers, Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications, August 30-September 03, 2004, Portland, Oregon, USA
• 5: N. Beheshti, Y. Ganjali, M. Ghobadi, N. McKeown, J. Naous, and G. Salmon. Time-sensitive network experiments. Technical Report TR08-SN-UT-04-08-00, University of Toronto, April 2008.
• 6: R. Bush , D. Meyer, Some Internet Architectural Guidelines and Philosophy, RFC Editor, 2002
• 7: Mark Carson , Darrin Santay, NIST Net: a Linux-based network emulation tool, ACM SIGCOMM Computer Communication Review, v.33 n.3, July 2003 [doi>10.1145/956993.957007]
• 8: Amogh Dhamdhere , Constantine Dovrolis, Open issues in router buffer sizing, ACM SIGCOMM Computer Communication Review, v.36 n.1, January 2006 [doi>10.1145/1111322.1111342]
• 9: M. Enachescu, Y. Ganjali, A. Goel, N. McKeown, and T. Roughgarden. Routers with very small buffers. In Proceedings of the IEEE Infocom, Barcelona, Spain, April 2006.
• 10: A. Erramilli, O. Narayan, A. Neidhardt, and I. Saniee. Performance impacts of multi-scaling in wide area TCP/IP traffic. In Proceedings of the IEEE Infocom, Tel-Aviv, Isreal, March 2000.
• 11: Anja Feldmann , Anna C. Gilbert , Polly Huang , Walter Willinger, Dynamics of IP traffic: a study of the role of variability and the impact of control, Proceedings of the conference on Applications, technologies, architectures, and protocols for computer communication, p.301-313, August 30-September 03, 1999, Cambridge, Massachusetts, United States
• 12: S. Floyd and E. Kohler. Internet research needs better models. In Proceedings of HotNets-I, October 2002.
• 13: C. Fraleigh, F. Tobagi, and C. Diot. Provisioning IP backbone networks to support latency sensitive traffic. In Proceedings of the IEEE Infocom, San Francisco, California, April 2003.
• 14: Charles Joseph Fraleigh , Fouad A. Tobagi, Provisioning internet backbone networks to support latency sensitive applications, Stanford University, Stanford, CA, 2002
• 15: Yashar Ganjali Gavgani , Nick Mckeown, Buffer sizing in internet routers, Stanford University, Stanford, CA, 2007
• 16: V. Jacobson. {e2e} re: Latest TCP measurements thoughts. Posting to the end-to-end mailing list, March 7, 1988.
• 17: V. Jacobson, Congestion avoidance and control, ACM SIGCOMM Computer Communication Review, v.18 n.4, p.314-329, August 1988
• 18: R. Morris, TCP behavior with many flows, Proceedings of the 1997 International Conference on Network Protocols (ICNP '97), p.205, October 28-31, 1997
• 19: R. Prasad, M. Jain, and C. Dovrolis. Effects of interrupt coalescence on network measurements. Passive and Active Measurements (PAM) conference, April 2004.
• 20: R. Prasad and M. K. Thottan. Inconsistencies with Spirent's TCP implementation, 2007.
• 21: R. S. Prasad and C. Dovrolis. Measuring the congestion responsiveness of Internet traffic. PAM, 2007.
• 22: Ravi S. Prasad , Constantine Dovrolis , Marina Thottan, Router buffer sizing revisited: the role of the output/input capacity ratio, Proceedings of the 2007 ACM CoNEXT conference, December 10-13, 2007, New York, New York [doi>10.1145/1364654.1364674]
• 23: G. Raina and D. Wischik. Buffer sizes for large multiplexers: TCP queueing theory and instability analysis. http://www.cs.ucl.ac.uk/staff/D.Wischik/Talks/tcptheory.html.
• 24: V. Ribeiro, R. Riedi, M. Crouse, and R. Baraniuk. Multiscale queuing analysis of long-range dependent network traffic. In Proceedings of the IEEE Infocom, Tel-Aviv, Isreal, March 2000.
• 25: Joel Sommers , Paul Barford, Self-configuring network traffic generation, Proceedings of the 4th ACM SIGCOMM conference on Internet measurement, October 25-27, 2004, Taormina, Sicily, Italy [doi>10.1145/1028788.1028798]
• 26: R. Takano, T. Kudoh, Y. Kodama, M. Matsuda, H. Tezuka, and Y. Ishikawa. Design and evaluation of precise software pacing mechanisms for fast long-distance networks. 3rd Intl. Workshop on Protocols for Fast Long-Distance Networks (PFLDnet), 2005.
• 27: Curtis Villamizar , Cheng Song, High performance TCP in ANSNET, ACM SIGCOMM Computer Communication Review, v.24 n.5, p.45-60, Oct. 1994 [doi>10.1145/205511.205520]
• 28: M. Wang and Y. Ganjali. The effects of fairness in buffer sizing. In Networking, pages 867--878, 2007.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.6 Internetworking

General Terms:
Experimentation, Measurement, Performance

Keywords:
netfpga, network test-beds, router buffer size, tcp