An integrated framework on mining logs files for computing system management

ABSTRACT

Traditional approaches to system management have been largely based on domain experts through a knowledge acquisition process that translates domain knowledge into operating rules and policies. This has been well known and experienced as a cumbersome, labor intensive, and error prone process. In addition, this process is difficult to keep up with the rapidly changing environments. In this paper, we will describe our research efforts on establishing an integrated framework for mining system log files for automatic management. In particular, we apply text mining techniques to categorize messages in log files into common situations, improve categorization accuracy by considering the temporal characteristics of log messages, develop temporal mining techniques to discover the relationships between different events, and utilize visualization tools to evaluate and validate the interesting temporal patterns for system management.

REFERENCES

• 1: Mark Berman. Testing for spatial association between a point process and another stochastic process. Applied Statistics, 35(1):54--62, 1986.
• 2: M. Chessell. Specification: Common base event, 2003. http://www-106.ibm.com /developerworks/webservices/library/ws-cbe/.
• 3: Noel A.C. Cressie. Statistics for spatial data. John Wiley & Sons, 1991.
• 4: Joseph L. Hellerstein, Sheng Ma, and Chang shing Perng. Discover actionable patterns in event data. IBM System Journal, 41(3):475--493, 2002.
• 5: K. Houck , S. Calo , A. Finkel, Towards a practical alarm correlation system, Proceedings of the fourth international symposium on Integrated network management IV, p.226-237, January 1995
• 6: Jeffrey O. Kephart , David M. Chess, The Vision of Autonomic Computing, Computer, v.36 n.1, p.41-50, January 2003 [doi>10.1109/MC.2003.1160055]
• 7: Nicholas Kushmerick, Edward Johnston, and Stephen McGuinness. Information extraction by text classification. Proceedings of the IJCAI-01 Workshop on Adaptive Text Extraction and Mining, 2001.
• 8: T. R. Leek. Information extraction using hidden markov models. Master's thesis, UC San Diego, 1997.
• 9: Cen Li , Gautam Biswas, Temporal Pattern Generation Using Hidden Markov Model Based Unsupervised Classification, Proceedings of the Third International Symposium on Advances in Intelligent Data Analysis, p.245-256, August 01, 1997
• 10: Tao Li , Sheng Ma, Mining Temporal Patterns Without Predefined Time Windows, Proceedings of the Fourth IEEE International Conference on Data Mining (ICDM'04), p.451-454, November 01-04, 2004
• 11: Feng Liang, Sheng Ma, and Joseph L. Hellerstein. Discovering fully dependent patterns. In SIAM DM, 2002.
• 12: Sheng Ma , Joseph L. Hellerstein, EventBrowser: A Flexible Tool for Scalable Analysis of Event Data, Proceedings of the 10th IFIP/IEEE International Workshop on Distributed Systems: Operations and Management: Active Technologies for Network and Service Management, p.285-296, October 11-13, 1999
• 13: Sheng Ma , Joseph L. Hellerstein, Mining Partially Periodic Event Patterns with Unknown Periods, Proceedings of the 17th International Conference on Data Engineering, p.205-214, April 02-06, 2001
• 14: Heikki Mannila, Hannu Toivonen, and A. Inkeri Verkamo. Discovering frequent episodes in sequences. In Proceedings of the First International Conference on Knowledge Discovery and Data Mining (SIGKDD'95), pages 210--215. AAAI Press, 1995.
• 15: A. McCallum and K. Nigam. A comparison of event models for naive Bayes text classification. In AAAI-98 Workshop on Learning for Text Categorization, 1998.
• 16: Thomas M. Mitchell, Machine Learning, McGraw-Hill Higher Education, 1997
• 17: Mining Logs Files for Computing System Management, Proceedings of the Second International Conference on Automatic Computing, p.309-310, June 13-16, 2005 [doi>10.1109/ICAC.2005.40]
• 18: L. R. Rabiner. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of IEEE, 77(2):257--286, 1989.
• 19: IBM Market Research. Autonomic computing core technology study, 2003.
• 20: Irina Rish. An empirical study of the naive Bayes classifier. In Proceedings of IJCAI-01 workshop on Empirical Methods in AI, pages 41--46, 2001.
• 21: Fabrizio Sebastiani, Machine learning in automated text categorization, ACM Computing Surveys (CSUR), v.34 n.1, p.1-47, March 2002 [doi>10.1145/505282.505283]
• 22: J. Stearley, Towards informatic analysis of syslogs, Proceedings of the 2004 IEEE International Conference on Cluster Computing, p.309-318, September 20-23, 2004
• 23: D. Stoyan, W.S. Kendall, and J. Mecke. Stochastic Geometry and its Applications. John wiley and Sons, 1995.
• 24: Brad Topol, David Ogle, Donna Pierson, Jim Thoensen, John Sweitzer, Marie Chow, Mary Ann Hoffmann, Pamela Durham, Ric Telford, Sulabha Sheth, and Thomas Studwell. Automating problem determination: A first step toward self-healing computing systems. IBM White Paper, October 2003. http://www-106.ibm.com/developerworks/autonomic/library/ac-summary/ac-prob.html.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning

Additional Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.4 Applications

General Terms:
Algorithms, Experimentation, Theory

Keywords:
event relationship, log categorization, system management, temporal pattern