Multivariate visualization in observation-based testing

ABSTRACT

We explore the use of multivariate visualization techniques to support a new approach to test data selection, called observation-based testing. Applications of multivariate visualization are described, including: evaluating and improving synthetic tests; filtering regression test suites; filtering captured operational executions; comparing test suites; and assessing bug reports. These applications are illustrated by the use of correspondence analysis to analyze test inputs for the GNU GCC compiler.

REFERENCES

• 1: Borg, I. and Groenen, P. Modern Multidimensional Scaling: Theory and Applications, Springer, 1997.
• 2: GCC. The GCC Home Page, http://www.gnu.org/software/gcc/gcc.html, Free Software Foundation, 1999.
• 3: Greenacre, M.J. Theory and Applications of Correspondence Analysis, Academic Press, 1984.
• 4: Stephen José Hanson , Robert E. Kraut , James M. Farber, Interface design and multivariate analysis of UNIX command use, ACM Transactions on Information Systems (TOIS), v.2 n.1, p.42-57, Jan. 1984 [doi>10.1145/357417.357421]
• 5: M. Jean Harrold , Rajiv Gupta , Mary Lou Soffa, A methodology for controlling the size of a test suite, ACM Transactions on Software Engineering and Methodology (TOSEM), v.2 n.3, p.270-285, July 1993 [doi>10.1145/152388.152391]
• 6: Mary Jean Harrold , Gregg Rothermel , Rui Wu , Liu Yi, An empirical investigation of program spectra, Proceedings of the 1998 ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering, p.83-90, June 16-16, 1998, Montreal, Quebec, Canada
• 7: Harville, D.A. Matrix Algebra From a Statistician's Perspective. Springer-Verlag, 1997.
• 8: W. J. Krzanowski, Principles of multivariate analysis: a user's perspective, Oxford University Press, Inc., New York, NY, 1988
• 9: Christina Pavlopoulou , Michal Young, Residual test coverage monitoring, Proceedings of the 21st international conference on Software engineering, p.277-284, May 16-22, 1999, Los Angeles, California, United States [doi>10.1145/302405.302637]
• 10: Andy Podgurski , Wassim Masri , Yolanda McCleese , Francis G. Wolff , Charles Yang, Estimation of software reliability by stratified sampling, ACM Transactions on Software Engineering and Methodology (TOSEM), v.8 n.3, p.263-283, July 1999 [doi>10.1145/310663.310667]
• 11: Andy Podgurski , Charles Yang, Partition testing, stratified sampling, and cluster analysis, Proceedings of the 1st ACM SIGSOFT symposium on Foundations of software engineering, p.169-181, December 08-10, 1993, Los Angeles, California, United States
• 12: Thomas Reps , Thomas Ball , Manuvir Das , James Larus, The use of program profiling for software maintenance with applications to the year 2000 problem, Proceedings of the 6th European conference held jointly with the 5th ACM SIGSOFT international symposium on Foundations of software engineering, p.432-449, September 22-25, 1997, Zurich, Switzerland
• 13: Gregg Rothermel , Mary Jean Harrold, A safe, efficient regression test selection technique, ACM Transactions on Software Engineering and Methodology (TOSEM), v.6 n.2, p.173-210, April 1997 [doi>10.1145/248233.248262]
• 14: X.Org. http://www.x.org/, X.Org, 1999.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

Additional Classification:
  G. Mathematics of Computing
  G.3 PROBABILITY AND STATISTICS
      Subjects: Multivariate statistics

  I. Computing Methodologies
  I.3 COMPUTER GRAPHICS

  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.3 Software Management
          Subjects: Software maintenance

General Terms:
Design, Management, Measurement, Performance, Reliability, Theory, Verification

Keywords:
correspondence analysis, data visualization, multivariate data analysis, multivariate visualization, observation-based testing, software testing