On the effective evaluation of TCP

ABSTRACT

Understanding the performance of the Internet's Transmission Control Protocol (TCP) is important because it is the dominant protocol used in the Internet today. Various testing methods exist to evaluate TCP performance, however all have pitfalls that need to be understood prior to obtaining useful results. Simulating TCP is difficult because of the wide range of variables, environments, and implementations available. Testing TCP modifications in the global Internet may not be the answer either: testing new protocols on real networks endangers other people's traffic and, if not done correctly, may also yield inaccurate or misleading results. In order for TCP research to be independently evaluated in the Internet research community there is a set of questions that researchers should try to answer. This paper attempts to list some of those questions and make recommendations as to how TCP testing can be structured to provide useful answers.

REFERENCES

• 1: Jong Suk Ahn , Peter B. Danzig , Zhen Liu , Limin Yan, Evaluation of TCP Vegas: emulation and experiment, Proceedings of the conference on Applications, technologies, architectures, and protocols for computer communication, p.185-195, August 28-September 01, 1995, Cambridge, Massachusetts, United States
• 2: {AFP98} Mark Allman, Sally Floyd, and Craig Partridge. Increasing TCP's Initial Window, September 1998. RFC 2414.
• 3: {AGS99} Mark Allman, Dan Glover, and Luis Sanchez. Enhancing TCP Over Satellite Channels Using Standard Mechanisms, January 1999. RFC 2488.
• 4: {A1197a} Mark Allman. Fixing Two BSD TCP Bugs. Technical Report CR-204151, NASA Lewis Research Center, October 1997.
• 5: {All97b} Mark Allman. Improving TCP Performance Over Satellite Channels. Master's thesis, Ohio University, June 1997.
• 6: Mark Allman, On the generation and use of TCP acknowledgments, ACM SIGCOMM Computer Communication Review, v.28 n.5, p.4-21, Oct. 1998 [doi>10.1145/303297.303301]
• 7: Mark Allman, TCP byte counting refinements, ACM SIGCOMM Computer Communication Review, v.29 n.3, July 1999 [doi>10.1145/505724.505727]
• 8: Mark Allman , Vern Paxson, On estimating end-to-end network path properties, Proceedings of the conference on Applications, technologies, architectures, and protocols for computer communication, p.263-274, August 30-September 03, 1999, Cambridge, Massachusetts, United States
• 9: {APS99} Mark Allman, Vern Paxson, and W. Richard Stevens. TCP Congestion Control, April 1999. RFC 2581.
• 10: {Arc} Internet Traffic Archive. http://ita.ee.lbl.gov.
• 11: {BCC+98} Robert Braden, David Clark, Jon Crowcroft, Bruce Davie, Steve Deering, Deborah Estrin, Sally Floyd, Van Jacobson, Greg Minshall, Craig Partridge, Larry Peterson, K. Ramakrishnan, S. Shenker, J. Wroclawski, and Lixia Zhang. Recommendations on Queue Management and Congestion Avoidance in the Internet, April 1998. RFC 2309.
• 12: {BLFN96} Tim Berners-Lee, R. Fielding, and H. Nielsen Hypertext Transfer Protocol --- HTTP/1.0, May 1996. RFC 1945.
• 13: {BP95} Lawrence Brakmo and Larry Peterson. TCP Vegas: End to End Congestion Avoidance on a Global Internet. IEEE Journal on Selected Areas in Communications, 13(8), October 1995.
• 14: Lawrence S. Brakmo , Larry L. Peterson, Experiences with network simulation, ACM SIGMETRICS Performance Evaluation Review, v.24 n.1, p.80-90, May 1996
• 15: {Bra89} Robert Braden. Requirements for Internet Hosts --- Communication Layers, October 1989. RFC 1122.
• 16: {DJ91} Peter Danzig and Sugih Jamin. tcplib: A Library of TCP/IP Traffic Characteristics. Technical Report CS-SYS-91-01, University of Southern California, October 1991.
• 17: {Fal99} Kevin Fall. Network Emulation in the Vint/NS Simulator. Technical report, University of California, Berkeley, January 1999.
• 18: {Fey85} Richard Feynman. Surely You're Joking Mr. Feynman!, chapter Cargo Cult Science, pages 308-317. Bantam, 1985.
• 19: Kevin Fall , Sally Floyd, Simulation-based comparisons of Tahoe, Reno and SACK TCP, ACM SIGCOMM Computer Communication Review, v.26 n.3, p.5-21, July 1996 [doi>10.1145/235160.235162]
• 20: {FGM+97} R. Fielding, Jim Gettys, Jeffrey C. Mogul, H. Frystyk, and Tim Berners-Lee. Hypertext Transfer Protocol --- HTTP/1.1, January 1997. RFC 2068.
• 21: Sally Floyd , Van Jacobson, Random early detection gateways for congestion avoidance, IEEE/ACM Transactions on Networking (TON), v.1 n.4, p.397-413, Aug. 1993 [doi>10.1109/90.251892]
• 22: Sally Floyd, TCP and explicit congestion notification, ACM SIGCOMM Computer Communication Review, v.24 n.5, p.8-23, Oct. 1994 [doi>10.1145/205511.205512]
• 23: {Flo99} Sally Floyd. Validation Experiences with the NS Simulator. Technical report, ACIRI, April 1999. White paper submitted to the DARPA Workshop on Validation of Large Scale Network Simulation Models.
• 24: John Heidemann, Performance interactions between P-HTTP and TCP implementations, ACM SIGCOMM Computer Communication Review, v.27 n.2, p.65-73, Apr. 1997 [doi>10.1145/263876.263886]
• 25: {Hey90} A. Heybey. The Network Simulator. Technical report, MIT, September 1990.
• 26: {HK99} Tom Henderson and Randy Katz. Transport Protocols for Internet-Compatible Satellite Networks. IEEE Journal on Selected Areas of Communications, 1999. To appear.
• 27: Janey C. Hoe, Improving the start-up behavior of a congestion control scheme for TCP, Conference proceedings on Applications, technologies, architectures, and protocols for computer communications, p.270-280, August 28-30, 1996, Palo Alto, California, United States
• 28: V. Jacobson, Congestion avoidance and control, Symposium proceedings on Communications architectures and protocols, p.314-329, August 16-18, 1988, Stanford, California, United States
• 29: {Jai91} Raj Jain. The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation and Modeling. Wiley, 1991.
• 30: {JBB92} Van Jacobson, Robert Braden, and David Borman. TCP Extensions for High Performance, May 1992. RFC 1323.
• 31: Srinivasan Keshav, REAL: A Network Simulator, University of California at Berkeley, Berkeley, CA, 1988
• 32: {KL86} B. Kantor and P. Lapsley. Network News Transfer Protocol: A Proposed Standard for the Stream-Based Transmission of News, February 1986. RFC 977.
• 33: {MA99} Matt Mathis and Mark Allman. Empirical Bulk Transfer Capacity, June 1999. Internet-Draft draft-ietf-ippm-btc-framework-01.txt.
• 34: {Mah99} Jamshid Mahdavi. Enabling High Performance Data Transfers on Hosts. Technical report, Pittsburgh Supercomputer Center, June 1999. http://www.psc.edu/networking/perf_tune.html.
• 35: {MD90} Jeffrey C. Mogul and Steve Deering. Path MTU Discovery, November 1990. RFC 1191.
• 36: {MDM96} Jack McCann, Steve Deering, and Jeffrey C. Mogul. Path MTU Discovery for IP version 6, August 1996. RFC 1981.
• 37: {MF95} Steven McCanne and Sally Floyd. NS (Network Simulator), 1995. URL http://www-nrg.ee.lbl.gov.
• 38: {MM96a} Matt Mathis and Jamshid Mahdavi. Diagnosing Internet Congestion with a Transport Layer Performance Tool. In Proceedings of INET, June 1996.
• 39: Matthew Mathis , Jamshid Mahdavi, Forward acknowledgement: refining TCP congestion control, Conference proceedings on Applications, technologies, architectures, and protocols for computer communications, p.281-291, August 28-30, 1996, Palo Alto, California, United States
• 40: {MM96c} Matt Mathis and Jamshid Mahdavi. TCP Rate-Halving with Bounding Parameters. Technical report, Pittsburgh Supercomputer Center, October 1996. URL: http://www.psc.edu/networking/papers/FACKnotes/current/.
• 41: {MMFR96} Matt Mathis, Jamshid Mahdavi, Sally Floyd, and Allyn Romanow. TCP Selective Acknowledgement Options, October 1996. RFC 2018.
• 42: Jeffrey C. Mogul, Observing TCP dynamics in real networks, Conference proceedings on Communications architectures & protocols, p.305-317, August 17-20, 1992, Baltimore, Maryland, United States
• 43: {MSML99} Matt Mathis, Jeff Semke, Jamshid Mahdavi, and Kevin Lahey. The Rate-Halving Algorithm for TCP Congestion Control, August 1999. Internet-Draft draft-mathis-tcp-ratehalving-00.txt (work in progress).
• 44: Matthew Mathis , Jeffrey Semke , Jamshid Mahdavi, The macroscopic behavior of the TCP congestion avoidance algorithm, ACM SIGCOMM Computer Communication Review, v.27 n.3, p.67-82, July 1997 [doi>10.1145/263932.264023]
• 45: {MSMV99} Greg Minshall, Yasushi Saito, Jeffrey C. Mogul, and Ben Verghese. Application Performance Pitfalls and TCP's Nagle Algorithm. In Workshop on Internet Server Performance, May 1999.
• 46: {Nag84} John Nagle. Congestion Control in IP/TCP Internetworks, January 1984. RFC 896.
• 47: {PAD+99} Vern Paxson, Mark Allman, Scott Dawson, William Fenner, Jim Griner, Ian Heavens, Kevin Lahey, Jeff Semke, and Bernie Volz. Known TCP Implementation Problems, March 1999. RFC 2525.
• 48: Vern Paxson, Automated packet trace analysis of TCP implementations, Proceedings of the ACM SIGCOMM '97 conference on Applications, technologies, architectures, and protocols for computer communication, p.167-179, September 14-18, 1997, Cannes, France
• 49: Vern Paxson, End-to-end Internet packet dynamics, Proceedings of the ACM SIGCOMM '97 conference on Applications, technologies, architectures, and protocols for computer communication, p.139-152, September 14-18, 1997, Cannes, France
• 50: {Pax97c} Vern Paxson. Measurements and Analysis of End-to-End Internet Dynamics. Ph.D. thesis, University of California Berkeley, 1997.
• 51: {PBS+96} Venkata Padmanabhan, Hari Balakrishnan, Keith Sklower, Elan Amir, and Randy Katz. Networking Using Direct Broadcast Saellite. In Proceedings of the First International Workshop on Satellite-based Information Services (WOSBIS), November 1996.
• 52: Vern Paxson , Sally Floyd, Wide area traffic: the failure of Poisson modeling, IEEE/ACM Transactions on Networking (TON), v.3 n.3, p.226-244, June 1995 [doi>10.1109/90.392383]
• 53: Vern Paxson , Sally Floyd, Why we don't know how to simulate the Internet, Proceedings of the 29th conference on Winter simulation, p.1037-1044, December 07-10, 1997, Atlanta, Georgia, United States [doi>10.1145/268437.268737]
• 54: {PMAM98} Vern Paxson, Jamshid Mahdavi, Andrew Adams, and Matt Mathis. An Architecture for Large-Scale Internet Measurement. IEEE Communications, 1998.
• 55: {Pos81} Jon Postel. Transmission Control Protocol, September 1981. RFC 793.
• 56: {PR85} Jon Postel and Joyce Reynolds. File Tranfer Protocol (FTP), October 1985. RFC 959.
• 57: {PS98} Steve Parker and Chris Schmechel. Some Testing Tools for TCP Implementors, August 1998. RFC 2398.
• 58: {RF99} K. K. Ramakrishnan and Sally Floyd. A Proposal to Add Explicit Congestion Notification (ECN) to IP, January 1999. RFC 2481.
• 59: {Ric99} Alan Richard. ISINT Performance Validation Test Report. Technical Report CR-209407, NASA Glenn Research Center, June 1999.
• 60: Luigi Rizzo, Dummynet: a simple approach to the evaluation of network protocols, ACM SIGCOMM Computer Communication Review, v.27 n.1, p.31-41, Jan. 1997 [doi>10.1145/251007.251012]
• 61: Jeffrey Semke , Jamshid Mahdavi , Matthew Mathis, Automatic TCP buffer tuning, Proceedings of the ACM SIGCOMM '98 conference on Applications, technologies, architectures, and protocols for computer communication, p.315-323, August 31-September 04, 1998, Vancouver, British Columbia, Canada
• 62: {Tec} Third Millennium Technologies. OpNet Modeler. http://www.mil3.com.
• 63: {TMW97} Kevin Thompson, Gregory Miller, and Rick Wilder. Wide-Area Internet Traffic Patterns and Characteristics. IEEE Network, 11(6):10-23, November/December 1997.
• 64: {VH97} Vikram Visweswaraiah and John Heidemann. Improving Restart of Idle TCP Connections. Technical Report 97-661, University of Southern California, August 1997.