A comparative study of pairwise regression techniques for problem determination

ABSTRACT

Many runtime metrics can be collected from modern software systems. Stable statistical relationships exist among these metrics. Deviation from these stable relationships indicates potential problems, allowing diagnosis of failures. There exist many modeling techniques to represent these relationships. However, which one to use is a question that has yet to be studied.

In this paper we compare the use of simple linear regression (SLR) to some of its more complex variants, including autoregressive regression and locally weighted regression. We consider the component coverage, model robustness, accuracy of diagnosis, and computation cost. Our study finds that while more flexible models can improve diagnosis accuracy, they achieve it at the cost of reduced robust-ness. In particular, we found the autoregressive regression model with exogenous input (ARX) to provide the most accurate diagnosis; however, it is the least robust of the techniques considered and the second most expensive. This study also finds that smoothing and other data transformations can noticeably improve results of SLR, thus providing an efficient alternative to ARX.

REFERENCES

• 1: Manoj Agarwal, Nikos Anerousis, Manish Gupta, Vijay Mann, Lily Mummert, and Narendran Sachindran. Problem determination in enterprise middleware systems using change point correlation of time series data. In Network Operations and Management Symposium, April 2006.
• 2: Christopher G. Atkeson , Andrew W. Moore , Stefan Schaal, Locally Weighted Learning, Artificial Intelligence Review, v.11 n.1-5, p.11-73, Feb. 1997 [doi>10.1023/A:1006559212014]
• 3: A. Brown, G. Kar, and A. Keller. An active approach to characterizing dynamic dependencies for problem determination in a distributed environment. In Integrated Network Management, pages 377--390, May 2001.
• 4: Joyce Coleman and Tony Lau. Set up and run a Trade6 benchmark with DB2 UDB. IBM developerWorks. http://www-128.ibm.com/developerworks/edu/-dm-dw-dm-0506lau.html?S_TACT=-105AGX11&S_CMP=LIB.
• 5: R. D. Cook and S. Weisberg. Residual and Influence in Regression. Chapman and Hall, New York, 1982.
• 6: Yixin Diao, Frank Eskesen, Steve Froehlich, Joseph L. Hellerstein, Alexander Keller, Lisa Spainhower, and Maheswaran Surendra. Generic on-line discovery of quantitative models for service level management. In Integrated Network Management, pages 157--170, 2003.
• 7: Zhen Guo , Guofei Jiang , Haifeng Chen , Kenji Yoshihira, Tracking Probabilistic Correlation of Monitoring Data for Fault Detection in Complex Systems, Proceedings of the International Conference on Dependable Systems and Networks, p.259-268, June 25-28, 2006 [doi>10.1109/DSN.2006.70]
• 8: Matthias Hauswirth , Peter F. Sweeney , Amer Diwan , Michael Hind, Vertical profiling: understanding the behavior of object-priented applications, Proceedings of the 19th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, October 24-28, 2004, Vancouver, BC, Canada
• 9: Joseph L. Hellerstein, Fan Zhang, and Perwez Shahabuddin. Characterizing normal operation of a web server: Application to workload forecasting and problem detection. In Proceedings of Computer Measurement Group, December 1998.
• 10: IBM Corporation. IBM WebSphere Application Server V6 Performance Tools. http://publib.boulder.ibm.com/-infocenter/ieduasst/v1r1m0/topic/com.-ibm.iea.wws_v6/was/6.0/Performance/-WASv6_PerformanceTools.pdf.
• 11: G. Jiang, H. Chen, and K. Yoshihira. Discovering likely invariants of distributed transaction systems for autonomic system management. In International Conference on Autonomic Computing, 2006.
• 12: Guofei Jiang , Haifeng Chen , Kenji Yoshihira, Modeling and Tracking of Transaction Flow Dynamics for Fault Detection in Complex Systems, IEEE Transactions on Dependable and Secure Computing, v.3 n.4, p.312-326, October 2006 [doi>10.1109/TDSC.2006.52]
• 13: Jeffrey O. Kephart , David M. Chess, The Vision of Autonomic Computing, Computer, v.36 n.1, p.41-50, January 2003 [doi>10.1109/MC.2003.1160055]
• 14: Lennart Ljung, System identification: theory for the user, Prentice-Hall, Inc., Upper Saddle River, NJ, 1986
• 15: Mohammad A. Munawar, Kevin Quan, and Paul A. S. Ward. Interaction analysis of heterogeneous monitoring data for autonomic problem determination. In IEEE International Symposium on Ubisafe Computing. IEEE Computer Society Press, 2007.
• 16: Mohammad A. Munawar and Paul A. S. Ward. Leveraging many simple statistical models to adaptively monitor software systems. In International Symposium on Parallel and Distributed Processing and Applications (ISPA), 2007.
• 17: Mohammad A. Munawar and Paul A. S. Ward. Adaptive monitoring in enterprise software systems. In Tackling Computer Systems Problems with Machine Learning Techniques (SysML), June 2006.
• 18: Ian H. Witten , Eibe Frank, Data Mining: Practical Machine Learning Tools and Techniques, Second Edition (Morgan Kaufmann Series in Data Management Systems), Morgan Kaufmann Publishers Inc., San Francisco, CA, 2005