Reference-driven performance anomaly identification

ABSTRACT

Complex system software allows a variety of execution conditions on system configurations and workload properties. This paper explores a principled use of reference executions--those of similar execution conditions from the target--to help identify the symptoms and causes of performance anomalies. First, to identify anomaly symptoms, we construct change profiles that probabilistically characterize expected performance deviations between target and reference executions. By synthesizing several single-parameter change profiles, we can scalably identify anomalous reference-to-target changes in a complex system with multiple execution parameters. Second, to narrow the scope of anomaly root cause analysis, we filter anomaly-related low-level system metrics as those that manifest very differently between target and reference executions. Our anomaly identification approach requires little expert knowledge or detailed models on system internals and consequently it can be easily deployed. Using empirical case studies on the Linux I/O subsystem and a J2EE-based distributed online service, we demonstrate our approach's effectiveness in identifying performance anomalies over a wide range of execution conditions as well as multiple system software versions. In particular, we discovered five previously unknown performance anomaly causes in the Linux 2.6.23 kernel. Additionally, our preliminary results suggest that online anomaly detection and system reconfiguration may help evade performance anomalies in complex online systems.

REFERENCES

• 1: Realistic nonstationary online workloads. http://www.cs.rochester.edu/u/stewart/models.html.
• 2: MySQL JDBC driver. http://www.mysql.com/products/connector.
• 3: R.A. Fisher. The arrangement of field experiments. J. of the Ministry of Agriculture of Great Britain, 33:503--513, 1926.
• 4: M. Grindal, J. Offutt, and S.F. Andler. Combination testing strategies: A survey. Software Testing, Verification and Reliability, 15(3):167--199, Mar. 2005.
• 5: Sitaram Iyer , Peter Druschel, Anticipatory scheduling: a disk scheduling framework to overcome deceptive idleness in synchronous I/O, Proceedings of the eighteenth ACM symposium on Operating systems principles, October 21-24, 2001, Banff, Alberta, Canada
• 6: Nikolai Joukov , Avishay Traeger , Rakesh Iyer , Charles P. Wright , Erez Zadok, Operating system profiling via latency analysis, Proceedings of the 7th symposium on Operating systems design and implementation, November 06-08, 2006, Seattle, Washington
• 7: Chuanpeng Li , Kai Shen, Managing prefetch memory for data-intensive online servers, Proceedings of the 4th conference on USENIX Conference on File and Storage Technologies, p.19-19, December 13-16, 2005, San Francisco, CA
• 8: Chuanpeng Li , Kai Shen , Athanasios E. Papathanasiou, Competitive prefetching for concurrent sequential I/O, Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007, March 21-23, 2007, Lisbon, Portugal
• 9: Linux kernel bug tracker. http://bugzilla.kernel.org/.
• 10: Linux kernel bug tracker on "many pre-mature anticipation timeouts in anticipatory I/O scheduler". http://bugzilla.kernel.org/show_bug.cgi?id=10756.
• 11: Michael P. Mesnier , Matthew Wachs , Raja R. Sambasivan , Alice X. Zheng , Gregory R. Ganger, Modeling the relative fitness of storage, Proceedings of the 2007 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, June 12-16, 2007, San Diego, California, USA
• 12: Patrick Reynolds , Charles Killian , Janet L. Wiener , Jeffrey C. Mogul , Mehul A. Shah , Amin Vahdat, Pip: detecting the unexpected in distributed systems, Proceedings of the 3rd conference on Networked Systems Design & Implementation, p.9-9, May 08-10, 2006, San Jose, CA
• 13: RUBiS: Rice University bidding system. http://rubis.objectweb.org.
• 14: Yossi Rubner , Carlo Tomasi , Leonidas J. Guibas, The Earth Mover's Distance as a Metric for Image Retrieval, International Journal of Computer Vision, v.40 n.2, p.99-121, Nov. 2000 [doi>10.1023/A:1026543900054]
• 15: Kai Shen , Ming Zhong , Chuanpeng Li, I/O system performance debugging using model-driven anomaly characterization, Proceedings of the 4th conference on USENIX Conference on File and Storage Technologies, p.23-23, December 13-16, 2005, San Francisco, CA
• 16: Christopher Stewart , Terence Kelly , Alex Zhang, Exploiting nonstationarity for performance prediction, Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007, March 21-23, 2007, Lisbon, Portugal
• 17: Christopher Stewart , Kai Shen, Performance modeling and system management for multi-component online services, Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation, p.71-84, May 02-04, 2005
• 18: Eno Thereska , Gregory R. Ganger, Ironmodel: robust performance models in the wild, Proceedings of the 2008 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, June 02-06, 2008, Annapolis, MD, USA
• 19: Avishay Traeger , Ivan Deras , Erez Zadok, DARC: dynamic analysis of root causes of latency distributions, Proceedings of the 2008 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, June 02-06, 2008, Annapolis, MD, USA
• 20: Joseph Tucek , Shan Lu , Chengdu Huang , Spiros Xanthos , Yuanyuan Zhou, Triage: diagnosing production run failures at the user's site, Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles, October 14-17, 2007, Stevenson, Washington, USA
• 21: Helen J. Wang , John C. Platt , Yu Chen , Ruyun Zhang , Yi-Min Wang, Automatic misconfiguration troubleshooting with peerpressure, Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation, p.17-17, December 06-08, 2004, San Francisco, CA
• 22: Andreas Zeller, Isolating cause-effect chains from computer programs, Proceedings of the 10th ACM SIGSOFT symposium on Foundations of software engineering, November 18-22, 2002, Charleston, South Carolina, USA [doi>10.1145/587051.587053]

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.8 Performance
          Subjects: Measurements

Additional Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.8 Performance
          Subjects: Modeling and prediction

General Terms:
Experimentation, Measurement, Performance, Reliability

Keywords:
operating system, performance anomaly