WYSIWYT testing in the spreadsheet paradigm: an empirical evaluation

ABSTRACT

Is it possible to achieve some of the benefits of formal testing within the informal programming conventions of the spreadsheet paradigm? We have been working on an approach that attempts to do so via the development of a testing methodology for this paradigm. Our “What You See Is What You Test” (WYSIWYT) methodology supplements the convention by which spreadsheets provide automatic immediate visual feedback about values by providing automatic immediate visual feedback about “testedness”. In previous work we described this methodology; in this paper, we present empirical data about the methodology's effectiveness. Our results show that the use of the methodology was associated with significant improvement in testing effectiveness and efficiency even with no training on the theory of testing or test adequacy that the model implements. These results may be due at least in part to the fact that use of the methodology was associated with a significant reduction in overconfidence.

REFERENCES

• 1: Allen L. Ambler , Margaret M. Burnett , Betsy A. Zimmerman, Operational versus definitional: a perspective on programming paradigms, Computer, v.25 n.9, p.28-43, September 1992 [doi>10.1109/2.156380]
• 2: F. Belli and O. Jack. A test coverage notion for logic programming. In The 6th Intl. Symp. Softw. Rel. Eng., pages 133{142, 1995.
• 3: Polly S. Brown , John D. Gould, An experimental study of people creating spreadsheets, ACM Transactions on Information Systems (TOIS), v.5 n.3, p.258-272, July 1987 [doi>10.1145/27641.28058]
• 4: Margaret M. Burnett , Herkimer J. Gottfried, Graphical definitions: expanding spreadsheet languages through direct manipulation and gestures, ACM Transactions on Computer-Human Interaction (TOCHI), v.5 n.1, p.1-33, March 1998 [doi>10.1145/274444.274445]
• 5: Ed Huai-hsin Chi , Joseph Konstan , Phillip Barry , John Riedl, A spreadsheet approach to information visualization, Proceedings of the 10th annual ACM symposium on User interface software and technology, p.79-80, October 14-17, 1997, Banff, Alberta, Canada [doi>10.1145/263407.263513]
• 6: C. Cook, K. Rothermel, M. Burnett, T. Adams, G. Rothermel, A. Sheretov, F. Cort, and J. Reichwein. Does Immediate Visual Feedback aboutTesting Aid Debugging in Spreadsheet Languages? Technical Report TR: 99-60-07, Oregon State University, Mar. 1999.
• 7: E. Duesterwald, R. Gupta, and M. L. Sofia. Rigorous data ow testing through output in uences. In Proc. 2nd Irvine Softw. Symp., Mar. 1992.
• 8: P. G. Frankl , S. N. Weiss, An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing, IEEE Transactions on Software Engineering, v.19 n.8, p.774-787, August 1993 [doi>10.1109/32.238581]
• 9: P. G. Frankl , E. J. Weyuker, An Applicable Family of Data Flow Testing Criteria, IEEE Transactions on Software Engineering, v.14 n.10, p.1483-1498, October 1988 [doi>10.1109/32.6194]
• 10: D. Gilmore. Interface design: Have we got it wrong? In K. Nordby, D. Gilmore, and S. Arnesen, editors, IN- TERACT'95. Chapman and Hall, London, 1995.
• 11: Monica Hutchins , Herb Foster , Tarak Goradia , Thomas Ostrand, Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria, Proceedings of the 16th international conference on Software engineering, p.191-200, May 16-21, 1994, Sorrento, Italy
• 12: W. Kuhn and A. U. Frank. The use of functional programming in the specification and testing process. In Intl. Conf. and Wkshp. Interoperating Geographic Info. Systems, Dec. 1997.
• 13: Jennifer L. Leopold , Allen L. Ambler, Keyboardless Visual Programming Using Voice, Handwriting, and Gesture, Proceedings of the 1997 IEEE Symposium on Visual Languages (VL '97), p.28, April 23-26, 1997
• 14: G. Luo, G. Bochmann, B. Sarikaya, and M. Boyer. Control- ow based testing of prolog programs. In The 3rd Intl. Symp. Softw. Rel. Eng., pages 104{113, 1992.
• 15: Martina Marré , Antonia Bertolino, Reducing and estimating the cost of test coverage criteria, Proceedings of the 18th international conference on Software engineering, p.486-494, March 25-29, 1996, Berlin, Germany
• 16: Brad A. Myers, Graphical techniques in a spreadsheet for specifying user interfaces, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.243-249, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108903]
• 17: F. Ouabdesselam and I. Parissis. Testing techniques for data- ow synchronous programs. In AADEBUG'95: 2nd Intl. Wkshp. Automated and Algorithmic Debugging, May 1995.
• 18: Raymond R. Panko, What we know about spreadsheet errors, Journal of End User Computing, v.10 n.2, p.15-21, Spring 1998
• 19: Sandra Rapps , Elaine J. Weyuker, Selecting software test data using data flow information, IEEE Transactions on Software Engineering, v.11 n.4, p.367-375, April 1985 [doi>10.1109/TSE.1985.232226]
• 20: Rothermel Gregg , Burnett Margaret , Li Lexin , DuPuis Christopher , Sheretov Andrei, A Methodology for Testing Spreadsheets, Oregon State University, Corvallis, OR, 1999
• 21: Gregg Rothermel , Mary Jean Harrold , Jeffery Ostrin , Christie Hong, An Empirical Study of the Effects of Minimization on the Fault Detection Capabilities of Test Suites, Proceedings of the International Conference on Software Maintenance, p.34, March 16-19, 1998
• 22: Gregg Rothermel , Lixin Li , Margaret Burnett, Testing Strategies for Form-Based Visual Programs, Proceedings of the Eighth International Symposium on Software Reliability Engineering (ISSRE '97), p.96, November 02-05, 1997
• 23: Gregg Rothermel , Lixin Li , Christopher DuPuis , Margaret Burnett, What you see is what you test: a methodology for testing form-based visual programs, Proceedings of the 20th international conference on Software engineering, p.198-207, April 19-25, 1998, Kyoto, Japan
• 24: Trevor J. Smedley , Philip T. Cox , Shannon L. Byrne, Expanding the utility of spreadsheets through the integration of visual programming and user interface objects, Proceedings of the workshop on Advanced visual interfaces, May 27-29, 1996, Gubbio, Italy [doi>10.1145/948449.948473]
• 25: Gunnvald B. Svendsen, The influence of interface style on problem solving, International Journal of Man-Machine Studies, v.35 n.3, p.379-397, Sept. 1991 [doi>10.1016/S0020-7373(05)80134-8]
• 26: G. Viehstaedt and A. Ambler. Visual representation and manipulation of matrices. J. Vis. Lang. and Comp., 3(3):273{298, Sept. 1992.
• 27: E. J. Weyuker, More Experience with Data Flow Testing, IEEE Transactions on Software Engineering, v.19 n.9, p.912-919, September 1993 [doi>10.1109/32.241773]
• 28: E. M. Wilcox , J. W. Atwood , M. M. Burnett , J. J. Cadiz , C. R. Cook, Does continuous visual feedback aid debugging in direct-manipulation programming systems?, Proceedings of the SIGCHI conference on Human factors in computing systems, p.258-265, March 22-27, 1997, Atlanta, Georgia, United States [doi>10.1145/258549.258721]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

Additional Classification:
  D. Software
  D.1 PROGRAMMING TECHNIQUES
  D.2 SOFTWARE ENGINEERING

  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.1 Office Automation
          Subjects: Spreadsheets

  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.3 Software Management
          Subjects: Software maintenance

General Terms:
Design, Experimentation, Management, Measurement, Performance, Reliability, Theory, Verification

Keywords:
empirical studies, spreadsheets, testing, visual programming