Strategies for sound internet measurement

ABSTRACT

Conducting an Internet measurement study in a sound fashion can be much more difficult than it might first appear. We present a number of strategies drawn from experiences for avoiding or overcoming some of the pitfalls. In particular, we discuss dealing with errors and inaccuracies; the importance of associating <i>meta-data</i> with measurements; the technique of calibrating measurements by examining outliers and testing for consistencies; difficulties that arise with large-scale measurements; the utility of developing a discipline for reliably reproducing analysis results; and issues with making datasets publicly available. We conclude with thoughts on the sorts of tools and community practices that can assist researchers with conducting sound measurement studies.

REFERENCES

• 1: M. Allman. Private communication. 2001.
• 2: M. Allman, E. Blanton, and W. Eddy. A Scalable System for Sharing Internet Measurements. Proc. PAM 2002.
• 3: Mark Allman , Wesley M. Eddy , Shawn Ostermann, Estimating loss rates with TCP, ACM SIGMETRICS Performance Evaluation Review, v.31 n.3, p.12-24, December 2003 [doi>10.1145/974036.974038]
• 4: P. Barford. Private communication. 2000.
• 5: Sally Floyd , Vern Paxson, Difficulties in simulating the internet, IEEE/ACM Transactions on Networking (TON), v.9 n.4, p.392-403, August 2001 [doi>10.1109/90.944338]
• 6: C. Fraleigh et al. Packet-Level Traffic Measurements from the Sprint IP Backbone. IEEE Network 17(6). 2003.
• 7: A. Gilat. MATLAB: An Introduction with Applications. Wiley Text Books. 2003.
• 8: M. Handley, C. Kreibich and V. Paxson. Network Intrusion Detection: Evasion, Traffic Normalization, and End-to-End Protocol Semantics. Proc. USENIX Security Symposium. 2001.
• 9: J. Heidemann, K. Mills, and S. Kumar. Expanding Confidence in Network Simulation. IEEE Network, 15(5):58--63. 2001.
• 10: Sharad Jaiswal , Gianluca Iannaccone , Christophe Diot , Jim Kurose , Don Towsley, Measurement and classification of out-of-sequence packets in a tier-1 IP backbone, Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment, November 06-08, 2002, Marseille, France [doi>10.1145/637201.637218]
• 11: E. Kohler. ipsumdump. http:www//icir.org/kohler/ipsumdump/. 2002.
• 12: C. Kreibich. Design and Implementation of Netdude, a Framework for Packet Trace Manipulation. Proc. USENIX/FREENIX, 2004.
• 13: P. Martin-Löf. The Notion of Redundancy and Its Use as a Quantitative Measure of the Discrepancy between a Statistical Hypothesis and a Set of Observational Data. Scandinavian Journal of Statistics, 1(1), pp. 3--18. 1974.
• 14: D. Mills. Network Time Protocol (Version 3): Specification, Implementation and Analysis. RFC 1305, Network Information Center, SRI International, Menlo Park, CA. March 1992.
• 15: D. Mills. Modelling and Analysis of Computer Network Clocks. Technical Report 92-5-2, Electrical Engineering Department, Univ. Delaware. mbox May 1992.
• 16: Greg Minshall. TCPdpriv manual. http:www.acm.org/sigcomm/ITA/. 1996.
• 17: J. Mogul. Trace anonymization misses the point. WWW 2002 panel on Web Measurements. http://www2002.org/presentations/mogul-n.pdf
• 18: S. Moon, P. Skelly and D. Towsley. Estimation and Removal of Clock Skew from Network Delay Measurements. Proc. IEEE INFOCOM. 1999.
• 19: Ruoming Pang , Vern Paxson, A high-level programming environment for packet trace anonymization and transformation, Proceedings of the 2003 conference on Applications, technologies, architectures, and protocols for computer communications, August 25-29, 2003, Karlsruhe, Germany [doi>10.1145/863955.863994]
• 20: Attila Pásztor , Darryl Veitch, PC based precision timing without GPS, Proceedings of the 2002 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, June 15-19, 2002, Marina Del Rey, California
• 21: Vern Paxson, Automated packet trace analysis of TCP implementations, Proceedings of the ACM SIGCOMM '97 conference on Applications, technologies, architectures, and protocols for computer communication, p.167-179, September 14-18, 1997, Cannes, France
• 22: Vern Edward Paxson, Measurements and analysis of end-to-end Internet dynamics, University of California at Berkeley, Berkeley, CA, 1998
• 23: Vern Paxson, On calibrating measurements of packet transit times, Proceedings of the 1998 ACM SIGMETRICS joint international conference on Measurement and modeling of computer systems, p.11-21, June 22-26, 1998, Madison, Wisconsin, United States
• 24: Vern Paxson, Bro: a system for detecting network intruders in real-time, Computer Networks: The International Journal of Computer and Telecommunications Networking, v.31 n.23-24, p.2435-2463, Dec. 1999 [doi>10.1016/S1389-1286(99)00112-7]
• 25: T. Ptacek and T. Newsham, Insertion, Evasion, and Denial of Service: Eluding Network Intrusion Detection. Technical report, Secure Networks. http:www.icir.org/vern/Ptacek-Newsham-Evasion-98.ps. Jan. 1998.
• 26: R Development Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. 2003.
• 27: W. Venables and B. Ripley. Modern Applied Statistics with S-PLUS. Springer. 1999.
• 28: Lan Wang , Xiaoliang Zhao , Dan Pei , Randy Bush , Daniel Massey , Allison Mankin , S. Felix Wu , Lixia Zhang, Observation and analysis of BGP behavior under stress, Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment, November 06-08, 2002, Marseille, France [doi>10.1145/637201.637231]
• 29: Jun Xu , Jinliang Fan , Mostafa H. Ammar , Sue B. Moon, Prefix-Preserving IP Address Anonymization: Measurement-Based Security Evaluation and a New Cryptography-Based Scheme, Proceedings of the 10th IEEE International Conference on Network Protocols, p.280-289, November 12-15, 2002
• 30: L. Zhang, Z. Liu, and C. Xia. Clock Synchronization Algorithms for Network Measurements. Proc. IEEE INFOCOM. 2002

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.5 Local and Wide-Area Networks
          Subjects: Internet (e.g., TCP/IP)

General Terms:
Experimentation, Measurement

Keywords:
calibration, datasets, internet measurement, meta-data, reproducibility