Broad-spectrum studies of log file analysis

ABSTRACT

This paper reports on research into applying the technique of log file analysis for checking test results to a broad range of testing and other tasks. The studies undertaken included applying log file analysis to both unit- and system-level testing and to requirements of both safety-critical and non-critical systems, and the use of log file analysis in combination with other testing methods. The paper also reports on the technique of using log file analyzers to simulate the software under test, both in order to validate the analyzers and to clarify requirements. It also discusses practical issues to do with the completeness of the approach, and includes comparisons to other recently-published approaches to log file analysis.

REFERENCES

• 1: Jean-Raymond Abrial, Steam-Boiler Control Specification Problem, Formal Methods for Industrial Applications, Specifying and Programming the Steam Boiler Control (the book grow out of a Dagstuhl Seminar, June 1995)., p.500-509, January 1996
• 2: J. Andrews, Testing using Log File Analysis: Tools, Methods, and Issues, Proceedings of the 13th IEEE international conference on Automated software engineering, p.157, October 13-16, 1998
• 3: J. H. Andrews. Theory and practice of log file analysis. Technical Report 524, Department of Computer Science, University of Western Ontario, May 1998.
• 4: J. H. Andrews. Testing using log file analysis: An extended example. In Electronic Commerce Technology Trends: Challenges and Opportunities. Midrange Computing, 1999. Proceedings of the First International Workshop on Technological Challenges of Electronic Commerce, Markham, Ont., Canada, September 1998.
• 5: J. H. Andrews , Nancy A. Day , Jeffrey J. Joyce, Using a Formal Description Technique to Model Aspects of a Global Air Traffic Telecommunications Network, Proceedings of the IFIP TC6 WG6.1 Joint International Conference on Formal Description Techniques for Distributed Systems and Communication Protocols (FORTE X) and Protocol Specification, Testing and Verification (PSTV XVII), p.417-432, November 18-21, 1997
• 6: Roong-Ko Doong , Phyllis G. Frankl, The ASTOOT approach to testing object-oriented programs, ACM Transactions on Software Engineering and Methodology (TOSEM), v.3 n.2, p.101-130, April 1994 [doi>10.1145/192218.192221]
• 7: J. W. Duran and S. C. Ntafos. An evaluation of random testing. IEEE Transactions on Software Engineering, 10(4):438{444, July 1984.
• 8: Martin S. Feather, Rapid application of lightweight formal methods for consistency analyses, IEEE Transactions on Software Engineering, v.24 n.11, p.948-959, November 1998
• 9: D. Heimbold and D. Luckham. Debugging Ada tasking programs. IEEE Software, 2(2):47{57, March 1985.
• 10: Daniel Hoffman , Paul Strooper, ClassBench: a framework for automated class testing, Software—Practice & Experience, v.27 n.5, p.573-597, May 1997 [doi>10.1002/(SICI)1097-024X(199705)27:5<573::AID-SPE98>3.3.CO;2-V]
• 11: ISO (International Organization for Standardization). ACSE Protocol, ITU-T Rec. X.227 - ISO/IEC 8650-1. International Organization for Standardization, 2nd edition, 1994.
• 12: Mehdi Jazayeri, Component Programming - a Fresh Look at Software Components, Proceedings of the 5th European Software Engineering Conference, p.457-478, September 25-28, 1995
• 13: J. Kundu , J. E. Cuny, A scalable, visual interface for debugging with event-based behavioral abstraction, Proceedings of the Fifth Symposium on the Frontiers of Massively Parallel Computation (Frontiers'95), p.472, February 06-09, 1995
• 14: T. Kunz, J. Black, D. Taylor, and T. Basten. Poet: Target-system-independent visualizations of complex distributed-application executions. The Computer Journal, 40(8):499{512, September 1997.
• 15: Nancy G. Leveson , Stephen S. Cha , Timothy J. Shimeall, Safety Verification of Ada Programs Using Software Fault Trees, IEEE Software, v.8 n.4, p.48-59, July 1991 [doi>10.1109/52.300036]
• 16: T. O. O'Malley, D.J.Richardson, and L. K. Dillon. E cient speci cation-based oracles for critical systems. In Proceedings of the California Software Symposium, 1996.
• 17: D. K. Peters and D. L. Parnas. Using test oracles generated from program documentation. In Proceedings of the International Symposium on Software Testing and Analysis, 1984.
• 18: S. Qiao and H. Zhang. An automatic logfile analyzer for parallel programs. In H. R. Arabnia, editor, Proceedings of the International Conference on Parallel and Distributed Processing Techniques and Applications, Vol. III, pages 1371-1376, Las Vegas, Nevada, USA, 1999.
• 19: Kuo-Chung Tai , Richard H. Carver , Evelyn E. Obaid, Debugging Concurrent Ada Programs by Deterministic Execution, IEEE Transactions on Software Engineering, v.17 n.1, p.45-63, January 1991 [doi>10.1109/32.67578]
• 20: Jeffrey Voas , Frank Charron , Gary McGraw , Keith Miller , Michael Friedman, Predicting How Badly "Good" Software Can Behave, IEEE Software, v.14 n.4, p.73-83, July 1997 [doi>10.1109/52.595959]
• 21: G. V. Bochmann , R. Dssouli , J. R. Zhao, Trace Analysis for Conformance and Arbitration Testing, IEEE Transactions on Software Engineering, v.15 n.11, p.1347-1356, November 1989 [doi>10.1109/32.41328]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
  D.3 PROGRAMMING LANGUAGES

  F. Theory of Computation
  F.1 COMPUTATION BY ABSTRACT DEVICES
      F.1.2 Modes of Computation
          Subjects: Relations among modes**

  I. Computing Methodologies
  I.6 SIMULATION AND MODELING

General Terms:
Design, Languages, Measurement, Performance, Reliability, Theory, Verification

Keywords:
lightweight formal methods, safety verification, specification, test oracles, testing