S3: a scalable sensing service for monitoring large networked systems

ABSTRACT

Efficiently operating and managing large scale distributed and federated systems is an extremely challenging problem. Current solutions are a combination of centralized management and significant over-provisioning of the infrastructure. With the explosion of new resource-intensive media applications and services, over provisioning of the infrastructure is no longer a viable option. Timely and accurate knowledge of the global environment (particularly the highly dynamic network path properties) is necessary for management of performance SLAs, just-in-time resource provisioning, near-optimal dynamic service placement and reuse, construction of network service overlays, and fast detection of failures and malicious attacks. Further, different applications require information about different aspects of the environment at different timescales. We propose S³, a Scalable Sensing Service, that achieves the above requirements and enables personalized sensing of the environment as dictated by applications.

REFERENCES

• 1: Appmanager: Planetlab application manager. http://appmanager.berkeley.intel-research.net/.
• 2: David D. Clark , Craig Partridge , J. Christopher Ramming , John T. Wroclawski, A knowledge plane for the internet, Proceedings of the 2003 conference on Applications, technologies, architectures, and protocols for computer communications, August 25-29, 2003, Karlsruhe, Germany [doi>10.1145/863955.863957]
• 3: C. Dovrolis, P. Ramanathan, and D. Moore. What do packet dispersion techniques measure? In Proceedings of the IEEE INFOCOM 2001, Anchorage, AK, April 2001.
• 4: FCAPS Overview: White paper. http://www.futsoft.com/pdf/fcapswp.pdf.
• 5: R. Fonseca, P. Sharma, S. Banerjee, S.-J. Lee, and S. Basu. Distributed querying of internet distance information. In Proceedings of the IEEE Global Internet 2005, Miami, FL, March 2005.
• 6: Paul Francis , Sugih Jamin , Cheng Jin , Yixin Jin , Danny Raz , Yuval Shavitt , Lixia Zhang, IDMaps: a global internet host distance estimation service, IEEE/ACM Transactions on Networking (TON), v.9 n.5, p.525-540, October 2001 [doi>10.1109/90.958323]
• 7: http://ganglia.sourceforge.net.
• 8: Krishna P. Gummadi , Stefan Saroiu , Steven D. Gribble, King: estimating latency between arbitrary internet end hosts, Proceedings of the 2nd ACM SIGCOMM Workshop on Internet measurment, November 06-08, 2002, Marseille, France [doi>10.1145/637201.637203]
• 9: J. M. Hellerstein, V. Paxson, L. Peterson, T. Roscoe, S. Shenker, and D. Wetherall. The Network Oracle. In Bulletin of the IEEE Computer Society Technical Committee on Data Engineering. 2005.
• 10: R. Huebsch, J. M. Hellerstein, N. Lanham, B. T. Loo, S. Shenker, and I. Stoica. Querying the Internet with PIER. In Proceedings of the VLDB Conference, May 2003.
• 11: Navendu Jain , Praveen Yalagandula , Mike Dahlin , Yin Zhang, INSIGHT: a distributed monitoring system for tracking continuous queries, Proceedings of the twentieth ACM symposium on Operating systems principles, October 23-26, 2005, Brighton, United Kingdom [doi>10.1145/1095810.1118611]
• 12: Ratul Mahajan , Neil Spring , David Wetherall , Thomas Anderson, User-level internet path diagnosis, Proceedings of the nineteenth ACM symposium on Operating systems principles, October 19-22, 2003, Bolton Landing, NY, USA
• 13: T. S. E. Ng and H. Zhang. Predicting Internet network distance with coordinates-based approaches. In Proceedings of the IEEE INFOCOM 2002.
• 14: V. N. Padmanabhan, S. Ramabhadran, and J. Padhye. NetProfiler: Profiling Wide-Area Networks Using Peer Cooperation. In IPTPS, Feb 2005.
• 15: G. Phillip B, B. Karp, Y. Ke, S. Nath, and S. Seshan. Irisnet: An architecture for a world-wide sensor web. In IEEE Pervasive Computing, Oct 2003.
• 16: M. Pias, J. Crowcroft, S. Wilbur, T. Harris, and S. Bhatti. Lighthouses for scalable distributed location. In Proceedings of the IPTPS 2003.
• 17: S. Ratnasamy, M. Handley, R. Karp, and S. Shenker. Topologically-aware overlay construction and server selection. In Proceedings of the IEEE INFOCOM 2002, June 2002.
• 18: V. Ribeiro, R. Riedi, R. Baraniuk, J. Navratil, and L. Cottrell. pathChirp: Efficient available bandwidth estimation for network paths. In Proceedings of the PAM 2003, La Jolla, CA, April 2003.
• 19: S. Saroiu. SProbe: A Fast Tool for Measuring Bottleneck Bandwidth in Uncooperative Environments.
• 20: Sridhar Srinivasan , Ellen Zegura, M-coop: A Scalable Infrastructure for Network Measurement, Proceedings of the The Third IEEE Workshop on Internet Applications, p.35, June 23-24, 2003
• 21: Ion Stoica , Robert Morris , David Karger , M. Frans Kaashoek , Hari Balakrishnan, Chord: A scalable peer-to-peer lookup service for internet applications, Proceedings of the 2001 conference on Applications, technologies, architectures, and protocols for computer communications, p.149-160, August 2001, San Diego, California, United States
• 22: Jacob Strauss , Dina Katabi , Frans Kaashoek, A measurement study of available bandwidth estimation tools, Proceedings of the 3rd ACM SIGCOMM conference on Internet measurement, October 27-29, 2003, Miami Beach, FL, USA [doi>10.1145/948205.948211]
• 23: Robbert Van Renesse , Kenneth P. Birman , Werner Vogels, Astrolabe: A robust and scalable technology for distributed system monitoring, management, and data mining, ACM Transactions on Computer Systems (TOCS), v.21 n.2, p.164-206, May 2003 [doi>10.1145/762483.762485]
• 24: Vxargs. http://dharma.cis.upenn.edu/planetlab/vxargs/.
• 25: H. Wang, J. Platt, Y. Chen, R. Zhang, and Y. Wang. Automatic Misconfiguration Troubleshooting with PeerPressure. In OSDI, Dec 2004.
• 26: Yi-Min Wang , Chad Verbowski , John Dunagan , Yu Chen , Helen J. Wang , Chun Yuan , Zheng Zhang, STRIDER: A Black-box, State-based Approach to Change and Configuration Management and Support, Proceedings of the 17th USENIX conference on System administration, October 26-31, 2003, San Diego, CA
• 27: M. Wawrzoniak, L. Peterson, and T. Roscoe. Sophia: An Information Plane for Networked Systems. In HotNets-II, 2003.
• 28: Bernard Wong , Aleksandrs Slivkins , Emin Gün Sirer, Meridian: a lightweight network location service without virtual coordinates, Proceedings of the 2005 conference on Applications, technologies, architectures, and protocols for computer communications, August 22-26, 2005, Philadelphia, Pennsylvania, USA
• 29: Z. Xu, P. Sharma, S.-J. Lee, and S. Banerjee. Netvigator: Scalable network proximity estimation. Technical report, HP Laboratories, 2005.
• 30: Yan Chen , David Bindel , Hanhee Song , Randy H. Katz, An algebraic approach to practical and scalable overlay network monitoring, Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications, August 30-September 03, 2004, Portland, Oregon, USA
• 31: Praveen Yalagandula , Mike Dahlin, A scalable distributed information management system, Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications, August 30-September 03, 2004, Portland, Oregon, USA

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.3 Network Operations
          Subjects: Network monitoring

Additional Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.4 Distributed Systems
          Subjects: Distributed applications
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Performance attributes

General Terms:
Design, Management, Performance