LOBSTER: a European platform for passive network traffic monitoring

ABSTRACT

Over the past few years we have been witnessing a large number of new programs and applications which generate prolific amounts of questionable, if not illegal, traffic that dominates our networks. Hoping from one port to another and using sophisticated encoding mechanisms, such applications have managed to evade traditional monitoring tools and confuse system administrators.

In this paper we present a concerted European effort to improve our understanding of the Internet through the LOBSTER passive network traffic monitoring infrastructure. By capitalizing on a novel Distributed Monitoring Application Programming Interface which enables the creation of sophisticated applications on top of commodity hardware, LOBSTER empowers a large number of researchers and system administrators into reaching a better understanding of the kind of traffic that flows through their networks.

We have been running LOBSTER for more than a year now and we have deployed close to forty sensors in twelve countries in three continents. Using LOBSTER sensors

• we have captured more than 600,000 sophisticated cyberattacks which attempted to masquerade themselves using advanced polymorphic approaches

• we have monitored the traffic of entire NRENs making it possible to identify the magnitude (as well as the sources) of file-sharing (peer to peer) traffic.

REFERENCES

• 1: WinPcap Remote Capture. http://www.winpcap.org/docs/docs31beta4/html/group__remote.html.
• 2: P. Akritidis, K. Anagnostakis, and E. Markatos. Efficient content-based detection of zero-day worms. Communications, 2005. ICC 2005. 2005 IEEE International Conference on, 2, 2005.
• 3: Sergio Andreozzi , Natascia De Bortoli , Sergio Fantinel , Antonia Ghiselli , Gian Luca Rubini , Gennaro Tortone , Maria Cristina Vistoli, GridICE: a monitoring service for Grid systems, Future Generation Computer Systems, v.21 n.4, p.559-571, April 2005 [doi>10.1016/j.future.2004.10.005]
• 4: D. Antoniades, M. Polychronakis, S. Antonatos, E. P. Markatos, S. Ubik, and A. Oslebo. Appmon: An application for accurate per application traffic characterization. In Proceedings of IST Broadband Europe 2006 Conference, December 2006.
• 5: P. Arlos, M. Fiedler, and A. A. Nilsson. A distributed passive measurement infrastructure. In Proceedings of the 6th International Passive and Active Network Measurement Workshop (PAM'05), pages 215--227, 2005.
• 6: Chuck Cranor , Theodore Johnson , Oliver Spataschek , Vladislav Shkapenyuk, Gigascope: a stream database for network applications, Proceedings of the 2003 ACM SIGMOD international conference on Management of data, June 09-12, 2003, San Diego, California [doi>10.1145/872757.872838]
• 7: C. Fraleigh, C. Diot, B. Lyles, S. Moon, P. Owezarski, D. Papagiannaki, and F. Tobagi. Design and Deployment of a Passive Monitoring Infrastructure. In Proceedings of the Passive and Active Measurement Workshop, Apr. 2001.
• 8: M. Grossglauser and J. Rexford. Passive traffic measurement for IP operations. In The Internet as a Large-Scale Complex System, pages 91--120. 2005.
• 9: G. Iannaccone, C. Diot, D. McAuley, A. Moore, I. Pratt, and L. Rizzo. The CoMo White Paper, 2004. http://como.intel-research.net/pubs/como.whitepaper.pdf.
• 10: D. Koukis, S. Antonatos, D. Antoniades, E. Markatos, and P. Trimintzios. A Generic Anonymization Framework for Network Traffic. Communications, 2006 IEEE International Conference on, 5, 2006.
• 11: S. Krishnan. rpcap. http://rpcap.sourceforge.net/.
• 12: M. L. Massie, B. N. Chun, and D. E. Culler. The Ganglia Distributed Monitoring System: Design, Implementation, and Experience. Parallel Computing, 30(7), July 2004.
• 13: S. McCanne, C. Leres, and V. Jacobson. libpcap. Lawrence Berkeley Laboratory, Berkeley, CA. (software available from http://www.tcpdump.org/).
• 14: D. Morato , E. Magana , M. Izal , J. Aracil , F. Naranjo , F. Astiz , U. Alonso , I. Csabai , P. Haga , G. Simon , J. Steger , G. Vattay, The European Traffic Observatory Measurement Infraestructure (ETOMIC): A Testbed for Universal Active and Passive Measurements, Proceedings of the First International Conference on Testbeds and Research Infrastructures for the DEvelopment of NeTworks and COMmunities, p.283-289, February 23-25, 2005 [doi>10.1109/TRIDNT.2005.34]
• 15: A. Papadogiannakis, A. Kapravelos, M. Polychronakis, E. P. Markatos, and A. Ciuffoletti. Passive end-to-end packet loss estimation for grid traffic monitoring. In Proceedings of the CoreGRID Integration Workshop, 2006.
• 16: Peter Morriessy. RMON2: To the Network Layer and Beyond! Network Computing, Feb. 1998. http://www.nwc.com/903/903f1.html.
• 17: M. Polychronakis, K. Anagnostakis, and E. Markatos. Network-level Polymorphic Shellcode Detection using Emulation. In Proceedings of the Third Conference on Detection of Intrusions and Malware and Vulnerability Assesment (DIMVA), 2006.
• 18: M. Polychronakis, K. Anagnostakis, and E. Markatos. Emulation-Based Detection of Non-self-contained Polymorphic Shellcode. In Proceedings of the 10th International Symposium on Recent Advances in Intrusion Detection (RAID), 2007.
• 19: M. Polychronakis, K. G. Anagnostakis, E. P. Markatos, and A. Øslebø. Design of an Application Programming Interface for IP Network Monitoring. In Proceedings of the 9th IFIP/IEEE Network Operations and Management Symposium (NOMS'04), pages 483--496, Apr. 2004.
• 20: J. Ritter. ngrep -- Network grep. http://ngrep.sourceforge.net/.
• 21: Martin Roesch, Snort - Lightweight Intrusion Detection for Networks, Proceedings of the 13th USENIX conference on System administration, November 07-12, 1999, Seattle, Washington
• 22: P. Trimintzios, M. Polychronakis, A. Papadogiannakis, M. Foukarakis, E. P. Markatos, and A. Øslebø. DiMAPI: An application programming interface for distributed network monitoring. In Proceedings of the 10^th IEEE/IFIP Network Operations and Management Symposium (NOMS), April 2006.
• 23: Sven Ubik , Demetres Antoniades , Arne Oslebo, ABW--Short-Timescale Passive Bandwidth Monitoring, Proceedings of the Sixth International Conference on Networking, p.49, April 22-28, 2007 [doi>10.1109/ICN.2007.15]
• 24: K. Wang, G. Cretu, and S. J. Stolfo. Anomalous payload-based worm detection and signature generation. In Proceedings of the 8th International Symposium on Recent Advances in Intrusion Detection (RAID), 2005.
• 25: J. Wu, S. Vangala, L. Gao, and K. Kwiat. An effective architecture and algorithm for detecting worms with various scan techniques. In Proceedings of the 11th Network and Distributed System Security Symposium (NDSS), 2004.
• 26: Cliff Changchun Zou , Lixin Gao , Weibo Gong , Don Towsley, Monitoring and early warning for internet worms, Proceedings of the 10th ACM conference on Computer and communications security, October 27-30, 2003, Washington D.C., USA [doi>10.1145/948109.948136]

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.3 Network Operations

Additional Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.0 General

General Terms:
Security

Keywords:
distributed monitoring, netork monitoring, traffic classification