Towards optimal sampling for flow size estimation

ABSTRACT

The flow size distribution is a useful metric for traffic modeling and management. It is well known however that its estimation based on sampled data is problematic. Previous work has shown that flow sampling (FS) offers enormous statistical benefits over packet sampling, however it suffers from high resource requirements and is not currently used in routers. In this paper we present Dual Sampling, which can to a large extent provide flow-sampling-like statistical performance for packet-sampling-like computational cost. Our work is grounded in a Fisher information based approach recently used to evaluate a number of sampling schemes, excluding however FS, for TCP flows. We show how to revise and extend the approach to include FS as well as DS and others, and how to make rigorous and fair comparisons. We show how DS significantly outperforms other packet based methods, but also prove that DS is inferior to flow sampling. However, since DS is a two-parameter family of methods which includes FS as a special case, DS can be used to approach flow sampling continuously. We then describe a packet sampling based implementation of DS and analyze its key computational costs to show that router implementation is feasible. Our approach offers insights into many issues, including how the notions of 'flow quality' and 'packet gain' can be used to understand the relative performance of methods, and how the problem of optimal sampling can be formulated. Our work is theoretical with some simulation support and a case study on Internet data.

REFERENCES

• 1: Nick Duffield , Carsten Lund , Mikkel Thorup, Estimating flow distributions from sampled flow statistics, IEEE/ACM Transactions on Networking (TON), v.13 n.5, p.933-946, October 2005 [doi>10.1109/TNET.2005.852874]
• 2: Nicolas Hohn , Darryl Veitch, Inverting sampled traffic, Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement, October 27-29, 2003, Miami Beach, FL, USA [doi>10.1145/948205.948235]
• 3: Nicolas Hohn , Darryl Veitch, Inverting sampled traffic, IEEE/ACM Transactions on Networking (TON), v.14 n.1, p.68-80, February 2006 [doi>10.1109/TNET.2005.863456]
• 4: Bruno Ribeiro , Don Towsley , Tao Ye , Jean C. Bolot, Fisher information of sampled packets: an application to flow size estimation, Proceedings of the 6th ACM SIGCOMM conference on Internet measurement, October 25-27, 2006, Rio de Janeriro, Brazil [doi>10.1145/1177080.1177083]
• 5: G. Varghese, Network Algorithmics. San Francicso: Elsevier/Morgan Kaufmann, 2005.
• 6: L. Yang and G. Michailidis, "Estimation of flow lengths from sampled traffic," in Proc. GLOBECOM, San Francisco, CA, Nov 2006.
• 7: Thomas M. Cover , Joy A. Thomas, Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing), Wiley-Interscience, 2006
• 8: A. Hero, J. Fessler, and M. Usman, "Exploring estimator bias-variance tradeoffs using the Uniform CR bound," IEEE Trans. Sig. Proc., vol. 44, no. 8, pp. 2026--2041, Aug 1996.
• 9: P. Tune and D. Veitch, "Fisher information in flow size distribution estimation: Technical report," Dept. E&EE, The University of Melbourne, Tech. Rep., 2008, available: http://www.ee.unimelb.edu.au/people/lsptune/index.html/.
• 10: J. D. Gorman and A. O. Hero, "Lower bounds for parametric estimation with constraints," IEEE Trans. Info. Th., vol. 26, no. 6, pp. 1285--1301, Nov 1990.
• 11: D. Harville, Matrix Algebra from a Statistician's Perspective. Springer-Verlag, 1997.
• 12: J. E. Strum, "Binomial matrices," The Two Year College Mathematics Journal, vol. 8, no. 5, pp. 260--266, November 1977.
• 13: R. Zamir, "A proof of the Fisher information inequality via a data processing argument," IEEE Trans. Info. Th., vol. 44, no. 3, pp. 1246--1250, May 1998.
• 14: E. L. Lehmann and G. Casella, Theory of Point Estimation, 2nd ed., ser. Springer Texts in Statistics. Springer, 1998.
• 15: R. T. Rockafellar, Convex Analysis, ser. Princeton Landmarks in Mathematics and Physics. Princeton University Press, 1970.
• 16: Cristian Estan , Ken Keys , David Moore , George Varghese, Building a better NetFlow, Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications, August 30-September 03, 2004, Portland, Oregon, USA
• 17: NLANR, Leipzig-II Trace Data, http://pma.nlanr.net/Special/leip2.html.
• 18: F. Zhang, Matrix Theory: Basic Results and Techniques. Springer-Verlag, 1999.
• 19: K. S. Miller, "On the inverse of the sum of matrices," Mathematics Magazine, vol. 54, no. 2, March 1981.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.1 Systems and Information Theory
          Subjects: Value of information

Additional Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.3 Network Operations
          Subjects: Network monitoring

  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Nonparametric statistics

General Terms:
Measurement, Theory

Keywords:
fisher information, flow size distribution, internet measurement, routers, sampling