Scaling issues in network monitoring

ABSTRACT

Monitoring a Tier-1 ISP's extremely large and diverse network places extreme scaling demands on a Data Stream Management System (DSMS). A typical application involves a large number stream queries executing on high volume streams, replicated at a large number of monitoring points, with the results fed to a stream warehouse supporting a large number of users. I will describe some of these challenges and how a DSMS and other tools have evolved to meet them.

REFERENCES

• 1: Chuck Cranor , Theodore Johnson , Oliver Spataschek , Vladislav Shkapenyuk, Gigascope: a stream database for network applications, Proceedings of the 2003 ACM SIGMOD international conference on Management of data, June 09-12, 2003, San Diego, California [doi>10.1145/872757.872838]
• 2: Graham Cormode , Theodore Johnson , Flip Korn , S. Muthukrishnan , Oliver Spatscheck , Divesh Srivastava, Holistic UDAFs at streaming speeds, Proceedings of the 2004 ACM SIGMOD international conference on Management of data, June 13-18, 2004, Paris, France [doi>10.1145/1007568.1007575]
• 3: Graham Cormode , Flip Korn , S. Muthukrishnan , Divesh Srivastava, Space- and time-efficient deterministic algorithms for biased quantiles over data streams, Proceedings of the twenty-fifth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, June 26-28, 2006, Chicago, IL, USA [doi>10.1145/1142351.1142389]
• 4: Lukasz Golab , Theodore Johnson , Oliver Spatscheck, Prefilter: predicate pushdown at streaming speeds, Proceedings of the 2nd international workshop on Scalable stream processing system, March 29-29, 2008, Nantes, France [doi>10.1145/1379272.1379280]
• 5: T. Johnson, S. Muthukrishnan, V. Shkapenyuk, O. Spatscheck: Query-Aware Sampling for Data Streams. SSPS 2007: 664--673
• 6: Theodore Johnson , Muthu S. Muthukrishnan , Vladislav Shkapenyuk , Oliver Spatscheck, Query-aware partitioning for monitoring massive network data streams, Proceedings of the 2008 ACM SIGMOD international conference on Management of data, June 09-12, 2008, Vancouver, Canada [doi>10.1145/1376616.1376730]
• 7: J. Li, K. Tufte, V. Shkapenyuk, V. Papadimos, T. Johnson, D. Maier. Out-of-Order Processing: A New Architecture for High-Performance Stream Systems. SIGMOD 2008.
• 8: Yijian Bai , Hetal Thakkar , Haixun Wang , Chang Luo , Carlo Zaniolo, A data stream language and system designed for power and extensibility, Proceedings of the 15th ACM international conference on Information and knowledge management, November 06-11, 2006, Arlington, Virginia, USA [doi>10.1145/1183614.1183664]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.4 Systems
          Subjects: Query processing

General Terms:
Design, Experimentation, Performance

Keywords:
data stream management system, network monitoring, scalability