An empirical study of fault localization for end-user programmers

ABSTRACT

End users develop more software than any other group of programmers, using software authoring devices such as e-mail filtering editors, by-demonstration macro builders, and spreadsheet environments. Despite this, there has been little research on finding ways to help these programmers with the dependability of their software. We have been addressing this problem in several ways, one of which includes supporting end-user debugging activities through fault localization techniques. This paper presents the results of an empirical study conducted in an end-user programming environment to examine the impact of two separate factors in fault localization techniques that affect technique effectiveness. Our results shed new insights into fault localization techniques for end-user programmers and the factors that affect them, with significant implications for the evaluation of those techniques.

REFERENCES

• 1: Robin Abraham , Martin Erwig, Header and Unit Inference for Spreadsheets Through Spatial Analyses, Proceedings of the 2004 IEEE Symposium on Visual Languages - Human Centric Computing (VLHCC'04), p.165-172, September 26-29, 2004 [doi>10.1109/VLHCC.2004.29]
• 2: H. Agrawal, J. Horgan, S. London, and W. Wong. Fault localization using execution slices and dataflow tests. In Proceedings of the Sixth IEEE International Symposium on Software Reliability Engineering, pages 143--151, Toulouse, France, October 1995.
• 3: Y. Ahmad, T. Antoniu, S. Goldwater, and S. Krishnamurthi. A type system for statically detecting spreadsheet errors. In Proceedings of the 18th IEEE International Conference on Automated Software Engineering, pages 174--183, Montreal, Quebec, Canada, October 2003.
• 4: C. Allwood. Error detection processes in statistical problem solving. Cognitive Science, 8(4):413--437, 1984.
• 5: Tudor Antoniu , Paul A. Steckler , Shriram Krishnamurthi , Erich Neuwirth , Matthias Felleisen, Validating the Unit Correctness of Spreadsheet Programs, Proceedings of the 26th International Conference on Software Engineering, p.439-448, May 23-28, 2004
• 6: Y. Ayalew and R. Mittermeir. Spreadsheet debugging. In Proceedings of the European Spreadsheet Risks Interest Group, Dublin, Ireland, July 2003.
• 7: Nicolas J. Belkin, Helping people find what they don't know, Communications of the ACM, v.43 n.8, p.58-61, Aug. 2000 [doi>10.1145/345124.345143]
• 8: Barry W. Boehm , Clark , Horowitz , Brown , Reifer , Chulani , Ray Madachy , Bert Steece, Software Cost Estimation with Cocomo II with Cdrom, Prentice Hall PTR, Upper Saddle River, NJ, 2000
• 9: Barry Boehm , Victor R. Basili, Software Defect Reduction Top 10 List, Computer, v.34 n.1, p.135-137, January 2001 [doi>10.1109/2.962984]
• 10: P. Bunus and P. Fritzson. Semi-automatic fault localization and behavior verification for physical system simulation models. In Proceedings of the 18th IEEE International Conference on Automated Software Engineering, pages 253--258, Montreal, Quebec, Canada, October 2003.
• 11: Margaret Burnett , John Atwood , Rebecca Walpole Djang , James Reichwein , Herkimer Gottfried , Sherry Yang, Forms/3: A first-order visual language to explore the boundaries of the spreadsheet paradigm, Journal of Functional Programming, v.11 n.2, p.155-206, March 2001
• 12: C. Corritore, B. Kracher, and S. Wiedenbeck. Trust in the online environment. In HCI International, volume 1, pages 1548--1552, New Orleans, Louisiana, USA, August 2001.
• 13: Richard A. DeMillo , Hsin Pan , Eugene H. Spafford, Critical slicing for software fault localization, Proceedings of the 1996 ACM SIGSOFT international symposium on Software testing and analysis, p.121-134, January 08-10, 1996, San Diego, California, United States
• 14: D. Hilzenrath. Finding errors a plus, Fannie says; mortgage giant tries to soften effect of $1 billion in mistakes. The Washington Post, October 31, 2003.
• 15: James A. Jones , Mary Jean Harrold , John Stasko, Visualization of test information to assist fault localization, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581397]
• 16: Andrew J. Ko , Brad A. Myers, Designing the whyline: a debugging interface for asking questions about program behavior, Proceedings of the SIGCHI conference on Human factors in computing systems, p.151-158, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985712]
• 17: J. Lyle and M. Weiser. Automatic program bug location by program slicing. In Proceedings of the 2nd International Conference on Computers and Applications, pages 877--883, 1987.
• 18: Robert C. Miller , Brad A. Myers, Outlier finding: focusing user attention on possible errors, Proceedings of the 14th annual ACM symposium on User interface software and technology, November 11-14, 2001, Orlando, Florida [doi>10.1145/502348.502361]
• 19: H. Pan and E. Spafford. Toward automatic localization of software faults. In Proceedings of the 10th Pacific Northwest Software Quality Conference, October 1992.
• 20: R. Panko. Finding spreadsheet errors: Most spreadsheet errors have design flaws that may lead to long-term miscalculation. Information Week, page 100, May 1995.
• 21: Raymond R. Panko, What we know about spreadsheet errors, Journal of End User Computing, v.10 n.2, p.15-21, Spring 1998
• 22: Orna Raz , Philip Koopman , Mary Shaw, Semantic anomaly detection in online data sources, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida [doi>10.1145/581339.581378]
• 23: M. Renieris and S. Reiss. Fault localization with nearest neighbor queries. In Proceedings of the 18th IEEE International Conference on Automated Software Engineering, pages 30--39, Montreal, Quebec, Canada, October 2003.
• 24: G. Robertson. Officials red-faced by $24m gaffe: Error in contract bid hits bottom line of TransAlta Corp. Ottawa Citizen, June 5, 2003.
• 25: Gregg Rothermel , Margaret Burnett , Lixin Li , Christopher Dupuis , Andrei Sheretov, A methodology for testing spreadsheets, ACM Transactions on Software Engineering and Methodology (TOSEM), v.10 n.1, p.110-147, Jan. 2001 [doi>10.1145/366378.366385]
• 26: J. Ruthruff , E. Creswick , M. Burnett , C. Cook , S. Prabhakararao , M. Fisher, II , M. Main, End-user software visualizations for fault localization, Proceedings of the 2003 ACM symposium on Software visualization, June 11-13, 2003, San Diego, California [doi>10.1145/774833.774851]
• 27: J. Ruthruff, S. Prabhakararao, J. Reichwein, C. Cook, E. Creswick, and M. Burnett. Interactive, visual fault localization support for end-user programmers. Journal of Visual Languages and Computing, 2005 (to appear).
• 28: J. Sajaniemi. Modeling spreadsheet audit: A rigorous approach to automatic visualization. Journal on Visual Languages and Computing, 11(1):49--82, February 2000.
• 29: S. Siegel and N. Castellan Jr. Non-parametric Statistics for the Behavioral Sciences. McGraw Hill, Boston, Massachusetts, USA, 1998.
• 30: F. Tip. A survey of program slicing techniques. Journal on Programming Languages, 3(3):121--189, 1995.
• 31: J. Voas. Software testability measurement for assertion placement and fault localization. In Proceedings of the International Workshop on Automated and Algorithmic Debugging, pages 133--144, 1995.
• 32: Earl J. Wagner , Henry Lieberman, Supporting user hypotheses in problem diagnosis, Proceedings of the 9th international conference on Intelligent user interface, January 13-16, 2004, Funchal, Madeira, Portugal [doi>10.1145/964442.964450]
• 33: Aaron Wilson , Margaret Burnett , Laura Beckwith , Orion Granatir , Ledah Casburn , Curtis Cook , Mike Durham , Gregg Rothermel, Harnessing curiosity to increase correctness in end-user programming, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642665]
• 34: Claes Wohlin , Per Runeson , Martin Höst , Magnus C. Ohlsson , Bjöorn Regnell , Anders Wesslén, Experimentation in software engineering: an introduction, Kluwer Academic Publishers, Norwell, MA, 2000

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Debugging aids

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)
      D.2.6 Programming Environments
          Subjects: Interactive environments

  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.1 Office Automation
          Subjects: Spreadsheets

General Terms:
Experimentation, Verification

Keywords:
debugging, end-user programming, end-user software engineering, fault localization