Streamsight: a visualization tool for large-scale streaming applications

ABSTRACT

Stream processing is becoming a new and important computing paradigm. Innovative streaming applications are being developed in areas ranging from scientific applications (e.g., environment monitoring), to business intelligence (e.g., fraud detection and trend analysis), to financial markets (e.g., algorithmic trading strategies). Developing, understanding, debugging, and optimizing streaming applications is non-trivial because of the adaptive and dynamic nature of these applications. The sheer complexity and the distributed character of a large number of cooperating components hosted on a distributed environment further complicate matters. In this paper we describe Streamsight, a new visualization tool built to examine, monitor, and help understand the dynamic behavior of streaming applications. Previously developed stream processing visualization tools focus solely on composition of dataflow graphs. Streamsight's novelty hinges on a wide range of capabilities, including the ability to manage the dynamics of large and evolving topologies comprising multiple streaming applications with thousands of nodes and interconnections. From rendering live performance counters using different perspectives to allowing recordings and replays of the execution process, Streamsight provides the mechanisms that permit a better understanding of the evolving and adaptive behavior of streaming applications. These capabilities are used for debugging purposes, for performance optimization, and management of resources, including capacity planning. More than 50 developers, both inside and outside IBM, have been using Streamsight.

REFERENCES

• 1: D. Abadi , D. Carney , U. Çetintemel , M. Cherniack , C. Convey , C. Erwin , E. Galvez , M. Hatoun , A. Maskey , A. Rasin , A. Singer , M. Stonebraker , N. Tatbul , Y. Xing , R. Yan , S. Zdonik, Aurora: a data stream management system, Proceedings of the 2003 ACM SIGMOD international conference on Management of data, June 09-12, 2003, San Diego, California [doi>10.1145/872757.872855]
• 2: {Abadi05} D. J. Abadi, Y. Ahmad, M. Balazinska, U. Çetintemel, M. Cherniack, J.-H. Hwang, W. Lindner, A. Maskey, A. Rasin, E. Ryvkina, N. Tatbul, Y. Xing, S. B. Zdonik, "The Design of the Borealis Stream Processing Engine", Proceedings of CIDR, 2005, Asilomar, CA
• 3: Arvind Arasu , Mitch Cherniack , Eduardo Galvez , David Maier , Anurag S. Maskey , Esther Ryvkina , Michael Stonebraker , Richard Tibbetts, Linear road: a stream data management benchmark, Proceedings of the Thirtieth international conference on Very large data bases, p.480-491, August 31-September 03, 2004, Toronto, Canada
• 4: Lisa Amini , Henrique Andrade , Ranjita Bhagwan , Frank Eskesen , Richard King , Philippe Selo , Yoonho Park , Chitra Venkatramani, SPC: a distributed, scalable platform for data mining, Proceedings of the 4th international workshop on Data mining standards, services and platforms, p.27-37, August 20-20, 2006, Philadelphia, Pennsylvania [doi>10.1145/1289612.1289615]
• 5: {DePauw01} W. De Pauw, E. Jensen, N. Mitchell, G. Sevitsky, J. M. Vlissides, J. Yang, "Visualizing the Execution of Java Programs", International Seminar Dagstuhl on Software Visualization, 2001, Germany
• 6: Wim De Pauw , Sophia Krasikov , John F. Morar, Execution patterns for visualizing web services, Proceedings of the 2006 ACM symposium on Software visualization, September 04-05, 2006, Brighton, United Kingdom [doi>10.1145/1148493.1148499]
• 7: Emden R. Gansner , Stephen C. North, An open graph visualization system and its applications to software engineering, Software—Practice & Experience, v.30 n.11, p.1203-1233, Sept. 2000 [doi>10.1002/1097-024X(200009)30:11<1203::AID-SPE338>3.3.CO;2-E]
• 8: Bugra Gedik , Henrique Andrade , Kun-Lung Wu , Philip S. Yu , Myungcheol Doo, SPADE: the system s declarative stream processing engine, Proceedings of the 2008 ACM SIGMOD international conference on Management of data, June 09-12, 2008, Vancouver, Canada [doi>10.1145/1376616.1376729]
• 9: {Görg04} C. Görg, P. Birke, M. Pohl, S. Diehl, "Dynamic Graph Drawing of Sequences of Orthogonal and Hierarchical Graphs". Proceedings of GD, 2004, New York, NY
• 10: Orla Greevy , Michele Lanza , Christoph Wysseier, Visualizing live software systems in 3D, Proceedings of the 2006 ACM symposium on Software visualization, September 04-05, 2006, Brighton, United Kingdom [doi>10.1145/1148493.1148501]
• 11: Navendu Jain , Lisa Amini , Henrique Andrade , Richard King , Yoonho Park , Philippe Selo , Chitra Venkatramani, Design, implementation, and evaluation of the linear road bnchmark on the stream processing core, Proceedings of the 2006 ACM SIGMOD international conference on Management of data, June 27-29, 2006, Chicago, IL, USA [doi>10.1145/1142473.1142522]
• 12: Laxmikant V. Kalé , Gengbin Zheng , Chee Wai Lee , Sameer Kumar, Scaling applications to massively parallel machines using Projections performance analysis tool, Future Generation Computer Systems, v.22 n.3, p.347-358, February 2006 [doi>10.1016/j.future.2004.11.020]
• 13: {Kaugars00} K. Kaugars, R. Zanny, E. de Doncker, "PARVIS: Visualizing Distributed Dynamic Partitioning Algorithms", Proceedings of PDPTA, 2000, Las Vegas, NV
• 14: {Kumar06} V. Kumar, Z. Cai, B. F. Cooper, G. Eisenhauer, K. Schwan, M. Mansour, B. Seshasayee, P. Widener, "IFLOW: Resource-Aware Overlays for Composing and Managing Distributed Information Flows", Proceedings of EuroSys, 2006, Leuven, Belgium
• 15: {Kunz97} T. Kunz, J. P. Black, D. J. Taylor, T. Basten, "Poet: Target-System Independent Visualizations of Complex Distributed-Application Executions", in the Computer Journal, 40(8): 499--512, 1997
• 16: Andrian Marcus , Louis Feng , Jonathan I. Maletic, 3D representations for software visualization, Proceedings of the 2003 ACM symposium on Software visualization, June 11-13, 2003, San Diego, California [doi>10.1145/774833.774837]
• 17: {Nagel96} W. E. Nagel, A. Arnold, M. Weber, H.-C. Hoppe, K. Solchenbach. "VAMPIR: Visualization and Analysis of MPI Resources", in Supercomputer 63, 12(1): 69--80, 1996
• 18: {Reyes04} J. C. Reyes, "A Graph Editing Framework for the StreamIt Language", Master's Thesis, Massachusetts Institute of Technology, 2004
• 19: Sameer S. Shende , Allen D. Malony, The Tau Parallel Performance System, International Journal of High Performance Computing Applications, v.20 n.2, p.287-311, May 2006 [doi>10.1177/1094342006064482]
• 20: {Streambase} "Streambase: Streambase System", retrieved from http://www.streambase.com, 2008
• 21: {Topol98} B. Topol, J. Stasko, V. Sunderam, "PVaniM: A Tool for Visualization in Network Computing Environments", in Concurrency: Practice & Experience, 10(14): 1197--1222, 1998
• 22: {Turaga07} D. S. Turaga, O. Verscheure, J. Wong, L. Amini, G. Yocum, E. Begle, B. Pfeifer, "Online FDC Control Limit Tuning with Yield Prediction using Incremental Decision Tree Learning", Sematech AEC/APC, 2007, Indian Wells, CA
• 23: {Wolf07} J. Wolf, N. Bansal, K. Hildrum, S. Parekh, D. Rajan, R. Wagle, K.-L. Wu, L. Fleischer, "SODA: An optimizing scheduler for large-scale stream-based distributed computer systems". Technical Report RC 24453, IBM Research, 2007
• 24: Kun-Lung Wu , Kirsten W. Hildrum , Wei Fan , Philip S. Yu , Charu C. Aggarwal , David A. George , Buǧra Gedik , Eric Bouillet , Xiaohui Gu , Gang Luo , Haixun Wang, Challenges and experience in prototyping a multi-modal stream analytic and monitoring application on System S, Proceedings of the 33rd international conference on Very large data bases, September 23-27, 2007, Vienna, Austria
• 25: G. J. Wills, Selection: 524,288 ways to say "this is interesting", Proceedings of the 1996 IEEE Symposium on Information Visualization (INFOVIS '96), p.54, October 28-29, 1996

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.6 Programming Environments

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Graphical user interfaces (GUI)

General Terms:
Design

Keywords:
distributed processing, dynamic visualization, stream processing visualization, visualizing topologies