What's going on?: learning communication rules in edge networks

ABSTRACT

Existing traffic analysis tools focus on traffic volume. They identify the heavy-hitters - flows that exchange high volumes of data, yet fail to identify the structure implicit in network traffic - do certain flows happen before, after or along with each other repeatedly over time? Since most traffic is generated by applications (web browsing, email, p2p), network traffic tends to be governed by a set of underlying rules. Malicious traffic such as network-wide scans for vulnerable hosts (mySQLbot) also presents distinct patterns.

We present eXpose, a technique to learn the underlying rules that govern communication over a network. From packet timing information, eXpose learns rules for network communication that may be spread across multiple hosts, protocols or applications. Our key contribution is a novel statistical rule mining technique to extract significant communication patterns in a packet trace without explicitly being told what to look for. Going beyond rules involving flow pairs, eXpose introduces templates to systematically abstract away parts of flows thereby capturing rules that are otherwise unidentifiable. Deployments within our lab and within a large enterprise show that eXpose discovers rules that help with network monitoring, diagnosis, and intrusion detection with few false positives.

REFERENCES

• 1: Paramvir Bahl , Ranveer Chandra , Albert Greenberg , Srikanth Kandula , David A. Maltz , Ming Zhang, Towards highly reliable enterprise network services via inference of multi-level dependencies, Proceedings of the 2007 conference on Applications, technologies, architectures, and protocols for computer communications, August 27-31, 2007, Kyoto, Japan
• 2: CoralReef - Workload Characterization. http://www.caida.org/analysis/workload/.
• 3: Roger Dingledine , Nick Mathewson , Paul Syverson, Tor: the second-generation onion router, Proceedings of the 13th conference on USENIX Security Symposium, p.21-21, August 09-13, 2004, San Diego, CA
• 4: Richard O. Duda , Peter E. Hart , David G. Stork, Pattern Classification (2nd Edition), Wiley-Interscience, 2000
• 5: Cristian Estan , Stefan Savage , George Varghese, Automatically inferring patterns of resource consumption in network traffic, Proceedings of the 2003 conference on Applications, technologies, architectures, and protocols for computer communications, August 25-29, 2003, Karlsruhe, Germany [doi>10.1145/863955.863972]
• 6: GAIM/Pidgin. http://www.pidgin.im/.
• 7: IDENT. http://www.grc.com/port_113.htm.
• 8: IPMON. http://ipmon.sprintlabs.com.
• 9: Jose Bernardo and Adrian F. M. Smith. Bayesian Theory. John Wiley, 2000.
• 10: Jayanthkumar Kannan , Jaeyeon Jung , Vern Paxson , Can Emre Koksal, Semi-automated discovery of application session structure, Proceedings of the 6th ACM SIGCOMM conference on Internet measurement, October 25-27, 2006, Rio de Janeriro, Brazil [doi>10.1145/1177080.1177096]
• 11: T. Karagiannis, K. Papagiannaki, and M. Faloutsos. BLINC: Multilevel Traffic Classification in the dark. In SIGCOMM, 2005.
• 12: Link-Level Multicast Name Resolution. http://www.windowsnetworking.com/articles_tutorials/Overview-Link-Local-Multicast-Name-Resolution.html.
• 13: PortPeeker Capture of mySQL Bot attack. http://www.linklogger.com/mySQLAttack.htm.
• 14: Nagios: Host, Service, Network Monitor. http://nagios.org.
• 15: T. Oetiker and D. Rand. Multi Router Traffic Grapher. http://people.ee.ethz.ch/~oetiker/webtools/mrtg/.
• 16: V. Paxson. Bro: A System For Detecting Network Intruders in Real-Time. Computer Networks, 1999.
• 17: Dave Plonka, FlowScan: A Network Traffic Flow Reporting and Visualization Tool, Proceedings of the 14th USENIX conference on System administration, December 03-08, 2000, New Orleans, Louisiana
• 18: Port 1081. http://isc.incidents.org/port.html?port=1081.
• 19: Patrick Reynolds , Charles Killian , Janet L. Wiener , Jeffrey C. Mogul , Mehul A. Shah , Amin Vahdat, Pip: detecting the unexpected in distributed systems, Proceedings of the 3rd conference on Networked Systems Design & Implementation, p.9-9, May 08-10, 2006, San Jose, CA
• 20: Analysis of the Sapphire Worm. http://www.caida.org/analysis/security/sapphire/.
• 21: P. Smyth and R. M. Goodman. Knowledge Discovery in Databases. MIT Press, 1991.
• 22: S. Staniford, S. Cheung, R. Crawford, M. Dilger, J. Frank, J. Hoagland, K. Levitt, C. Wee, R. Yip, and D. Zerkle. GrIDS: A Graph-based Intrusion Detection System for Large Networks. In National Information Systems Security Conference, 1996.
• 23: R. Kannan , S. Vempala , A. Veta, On clusterings-good, bad and spectral, Proceedings of the 41st Annual Symposium on Foundations of Computer Science, p.367, November 12-14, 2000
• 24: S. Venkataraman, D. Song, P. Gibbons, and A. Blum. New Streaming Algorithms for Fast Detection of Superspreaders. In NDSS, 2005.
• 25: Kunikazu Yoda , Hiroaki Etoh, Finding a Connection Chain for Tracing Intruders, Proceedings of the 6th European Symposium on Research in Computer Security, p.191-205, October 04-06, 2000
• 26: Yin Zhang , Vern Paxson, Detecting stepping stones, Proceedings of the 9th conference on USENIX Security Symposium, p.13-13, August 14-17, 2000, Denver, Colorado

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.2 Network Protocols

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Correlation and regression analysis

General Terms:
Algorithms, Design, Management, Measurement, Performance

Keywords:
communication rules, correlation, expose, rule mining