An empirical evaluation of a testing and debugging methodology for Excel

ABSTRACT

Spreadsheets are one of the most commonly used types of programs in the world, and it is important that they be sufficiently dependable. To help end users who create spreadsheets do so more reliably, we have created a testing and debugging methodology and environment for use in spreadsheets, known as the WYSIWYT methodology. Our prior experiments with WYSIWYT show that users can utilize it to ensure that their spreadsheets are more dependable, but these experiments to date have considered only an unfamiliar prototype spreadsheet environment, and have not involved spreadsheet creation tasks. In this work we conducted a controlled experiment that addresses these limitations. The results of this study indicate that the use of WYSIWYT did not affect the correctness of spreadsheets created by users, but it did significantly reduce the amount of effort required to create them. Further, the subjects' evaluation of the help provided by WYSIWYT was very positive. Our results provide several insights into the use of the WYSIWYT methodology by end users.

REFERENCES

• 1: Robin Abraham , Martin Erwig, Header and Unit Inference for Spreadsheets Through Spatial Analyses, Proceedings of the 2004 IEEE Symposium on Visual Languages - Human Centric Computing (VLHCC'04), p.165-172, September 26-29, 2004 [doi>10.1109/VLHCC.2004.29]
• 2: Ahmad, Y., Antoniu, T., Goldwater, S., and Krishnamurthi, S. "A Type System for Statically Detecting Spreadsheet Errors". In Proceedings of International Conference on Automated Software Engineering. Oct., 2003 p. 174--183
• 3: Tudor Antoniu , Paul A. Steckler , Shriram Krishnamurthi , Erich Neuwirth , Matthias Felleisen, Validating the Unit Correctness of Spreadsheet Programs, Proceedings of the 26th International Conference on Software Engineering, p.439-448, May 23-28, 2004
• 4: Ayalew, Y. and Mittermeir, R. "Interval-Based Testing for Spreadsheets". In Proceedings of International Arab Conference on Information Technology. University of Qatar, Qatar. Dec., 2002 p. 414--422
• 5: Laura Beckwith , Margaret Burnett , Curtis Cook, Reasoning about Many-to-Many Requirement Relationships in Spreadsheets, Proceedings of the IEEE 2002 Symposia on Human Centric Computing Languages and Environments (HCC'02), p.149, September 03-06, 2002
• 6: Laura Beckwith , Margaret Burnett , Susan Wiedenbeck , Curtis Cook , Shraddha Sorte , Michelle Hastings, Effectiveness of end-user debugging software features: are there gender issues?, Proceedings of the SIGCHI conference on Human factors in computing systems, April 02-07, 2005, Portland, Oregon, USA [doi>10.1145/1054972.1055094]
• 7: Margaret Burnett , John Atwood , Rebecca Walpole Djang , James Reichwein , Herkimer Gottfried , Sherry Yang, Forms/3: A first-order visual language to explore the boundaries of the spreadsheet paradigm, Journal of Functional Programming, v.11 n.2, p.155-206, March 2001
• 8: Margaret Burnett , Andrei Sheretov , Bing Ren , Gregg Rothermel, Testing Homogeneous Spreadsheet Grids with the "What You See Is What You Test" Methodology, IEEE Transactions on Software Engineering, v.28 n.6, p.576-594, June 2002 [doi>10.1109/TSE.2002.1010060]
• 9: Margaret Burnett , Curtis Cook , Omkar Pendse , Gregg Rothermel , Jay Summet , Chris Wallace, End-user software engineering with assertions in the spreadsheet paradigm, Proceedings of the 25th International Conference on Software Engineering, May 03-10, 2003, Portland, Oregon
• 10: Markus Clermont, Analyzing Large Spreadsheet Programs, Proceedings of the 10th Working Conference on Reverse Engineering, p.306, November 13-17, 2003
• 11: Clermont, M. and Mittermeir, R. "Auditing Large Spreadsheet Programs". In Proceedings of International Conference on Information Systems Implementation and Modelling. Apr., 2003 p. 87--97
• 12: Cullen, D., Excel Snafu Costs Firm $24 Million, in The Register. 2003.
• 13: Martin Erwig , Robin Abraham , Irene Cooperstein , Steve Kollmansberger, Automatic generation and maintenance of correct spreadsheets, Proceedings of the 27th international conference on Software engineering, p.136-145, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062494]
• 14: Marc Fisher , Mingming Cao , Gregg Rothermel , Curtis R. Cook , Margaret M. Burnett, Automated test case generation for spreadsheets, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581359]
• 15: Marc Fisher II , Dalai Jin , Gregg Rothermel , Margaret Burnett, Test Reuse in the Spreadsheet Paradigm, Proceedings of the 13th International Symposium on Software Reliability Engineering (ISSRE'02), p.257, November 12-15, 2002
• 16: Fisher II, M., Rothermel, G., Creelan, T., and Burnett, M. Scaling a Dataflow Testing Methodology to the Multiparadigm World of Commercial Spreadsheets. TR-UNL-CSE-2005-0003. University of Nebraska -- Lincoln: Lincoln, NE USA.2005.
• 17: Andrew J. Ko , Brad A. Myers, Designing the whyline: a debugging interface for asking questions about program behavior, Proceedings of the SIGCHI conference on Human factors in computing systems, p.151-158, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985712]
• 18: Incorporating Incremental Validation and Impact Analysis into Spreadsheet Maintenance: An Empirical Study, Proceedings of the IEEE International Conference on Software Maintenance (ICSM'01), p.72, November 07-09, 2001
• 19: Raymond R. Panko , Richard P. Halverson Jr, Spreadsheets on Trial: A Survey of Research on Spreadsheet Risks, Proceedings of the 29th Hawaii International Conference on System Sciences (HICSS) Volume 2: Decision Support and Knowledge-Based Systems, p.326, January 03-06, 1996
• 20: Raymond R. Panko, What we know about spreadsheet errors, Journal of End User Computing, v.10 n.2, p.15-21, Spring 1998
• 21: James Reichwein , Gregg Rothermel , Margaret Burnett, Slicing spreadsheets: an integrated methodology for spreadsheet testing and debugging, Proceedings of the 2nd conference on Domain-specific languages, p.25-38, October 03-06, 1999, Austin, Texas, United States
• 22: Gregg Rothermel , Lixin Li , Margaret Burnett, Testing Strategies for Form-Based Visual Programs, Proceedings of the Eighth International Symposium on Software Reliability Engineering (ISSRE '97), p.96, November 02-05, 1997
• 23: Gregg Rothermel , Margaret Burnett , Lixin Li , Christopher Dupuis , Andrei Sheretov, A methodology for testing spreadsheets, ACM Transactions on Software Engineering and Methodology (TOSEM), v.10 n.1, p.110-147, Jan. 2001 [doi>10.1145/366378.366385]
• 24: Karen J. Rothermel , Curtis R. Cook , Margaret M. Burnett , Justin Schonfeld , T. R. G. Green , Gregg Rothermel, WYSIWYT testing in the spreadsheet paradigm: an empirical evaluation, Proceedings of the 22nd international conference on Software engineering, p.230-239, June 04-11, 2000, Limerick, Ireland [doi>10.1145/337180.337206]
• 25: J. Ruthruff , E. Creswick , M. Burnett , C. Cook , S. Prabhakararao , M. Fisher, II , M. Main, End-user software visualizations for fault localization, Proceedings of the 2003 ACM symposium on Software visualization, June 11-13, 2003, San Diego, California [doi>10.1145/774833.774851]
• 26: Joseph R. Ruthruff , Margaret Burnett , Gregg Rothermel, An empirical study of fault localization for end-user programmers, Proceedings of the 27th international conference on Software engineering, p.352-361, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062523]
• 27: Sajaniemi, J., Modeling Spreadsheet Audit: A Rigorous Approach to Automatic Visualization. Journal of Visual Languages and Computing, 2000. 11(1): p. 49--82.
• 28: Scott, A., Shurgard Stock Dives after Auditor Quits over Company's Accounting, in The Seattle Times. 2003.
• 29: Susan Wiedenbeck , Patti L. Zila , Daniel S. McConnell, End-user training: an empirical study comparing on-line practice methods, Proceedings of the SIGCHI conference on Human factors in computing systems, p.74-81, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223914]
• 30: Aaron Wilson , Margaret Burnett , Laura Beckwith , Orion Granatir , Ledah Casburn , Curtis Cook , Mike Durham , Gregg Rothermel, Harnessing curiosity to increase correctness in end-user programming, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642665]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

Additional Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.1 Office Automation
          Subjects: Spreadsheets

Keywords:
empirical study, end-user software engineering, human subjects