Investigation of failure causes in workload-driven reliability testing

ABSTRACT

Virtual execution environments and middleware are required to be extremely reliable because applications running on top of them are developed assuming their correctness, and platform-level failures can result in serious and unexpected application-level problems. Since software platforms and middleware are often executed for long time without any interruption, large part of the testing process is devoted to investigate their behavior when long and stressful executions occur (these test cases are called workloads). When a problem is identified, software engineers examine log files to find its root cause. Unfortunately, since of the workloads length, log files can contain a huge amount of information and manual analysis is often prohibitive. Thus, de-facto, the identification of the root cause is mostly left to the intuition of the software engineer.

In this paper, we propose a technique to automatically analyze logs obtained from workloads to retrieve important information that can relate the failure to its cause. The technique works in three steps: (1) during workload executions, the system under test is monitored; (2) logs extracted from workloads that have been successfully completed are used to derive compact and general models of the expected behavior of the target system; (3) logs corresponding to workloads terminated unsuccessfully are compared with the inferred models to identify anomalous event sequences. Anomalies help software engineers to identify failure causes. The technique can also be used during operational phase, to discover possible causes of unexpected failures by comparing logs corresponding to failing executions with models derived at testing time. Preliminary experimental results conducted on the Java Virtual Machine indicate that several bugs can be rapidly identified thanks to the feedbacks provided by our technique.

REFERENCES

• 1: J. Andrews, Testing using Log File Analysis: Tools, Methods, and Issues, Proceedings of the 13th IEEE international conference on Automated software engineering, p.157, October 13-16, 1998
• 2: Apache Software Foundation. James server. http://james.apache.org/.
• 3: Domenico Cotroneo , Salvatore Orlando , Stefano Russo, Failure classification and analysis of the Java Virtual Machine, Proceedings of the 26th IEEE International Conference on Distributed Computing Systems, p.17, July 04-07, 2006 [doi>10.1109/ICDCS.2006.37]
• 4: Jason Gait, A probe effect in concurrent programs, Software—Practice & Experience, v.16 n.3, p.225-233, March 1986 [doi>10.1002/spe.4380160304]
• 5: Jiawei Han, Data Mining: Concepts and Techniques, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2005
• 6: IBM. Jikes Research Virtual Machine. http://jikesrvm.sourceforge.net.
• 7: Java Community Process (JCP). JSR-163: Java Platform Profiling Architecture (JPPA), 2004.
• 8: James Gosling , Bill Joy , Guy L. Steele, The Java Language Specification, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1996
• 9: I. Lee, R. Iyer, and D. Tang. Error/failure analysis using event logs from fault tolerant systems. In Proceedings of the 21st International Symposium on Fault-Tolerant Computing, 1991. (FTCS'21), pages 10--17, 1991.
• 10: T. Lin and D. Siewiorek. Error log analysis: statistical modeling and heuristic trend analysis. IEEE Transactions on Reliability, 39(4):419 -- 432, 1989.
• 11: Tim Lindholm , Frank Yellin, Java Virtual Machine Specification, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1999
• 12: Leonardo Mariani , Mauro Pezzè, Dynamic Detection of COTS Component Incompatibility, IEEE Software, v.24 n.5, p.76-85, September 2007 [doi>10.1109/MS.2007.138]
• 13: Charles E. McDowell , David P. Helmbold, Debugging concurrent programs, ACM Computing Surveys (CSUR), v.21 n.4, p.593-622, Dec. 1989 [doi>10.1145/76894.76897]
• 14: Salvatore Orlando , Stefano Russo, Java Virtual Machine Monitoring for Dependability Benchmarking, Proceedings of the Ninth IEEE International Symposium on Object and Component-Oriented Real-Time Distributed Computing, p.433-440, April 24-26, 2006 [doi>10.1109/ISORC.2006.45]
• 15: G. Pintér, H. Madeira, M. Vieira, I. Majzik, and A. Pataricza. A data mining approach to identify key factors in dependability experiments. In Proceedings of the 5th European Dependable Computing Conference (EDCC '05), pages 263--280, 2005.
• 16: Daniel P. Siewiorek , Ram Chillarege , Zbigniew T. Kalbarczyk, Reflections on Industry Trends and Experimental Research in Dependability, IEEE Transactions on Dependable and Secure Computing, v.1 n.2, p.109-127, April 2004 [doi>10.1109/TDSC.2004.20]
• 17: James E. Smith , Ravi Nair, The Architecture of Virtual Machines, Computer, v.38 n.5, p.32-38, May 2005 [doi>10.1109/MC.2005.173]
• 18: Sun. Hotspot Java Virtual Machine. http://java.sun.com/products/hotspot/.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Monitors

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Tracing

General Terms:
Reliability

Keywords:
JVM monitoring, automated analysis, log file analysis, model inference, workloads execution