What makes a good bug report?

ABSTRACT

In software development, bug reports provide crucial information to developers. However, these reports widely differ in their quality. We conducted a survey among developers and users of APACHE, ECLIPSE, and MOZILLA to find out what makes a good bug report.

The analysis of the 466 responses revealed an information mismatch between what developers need and what users supply. Most developers consider steps to reproduce, stack traces, and test cases as helpful, which are at the same time most difficult to provide for users. Such insight is helpful to design new bug tracking tools that guide users at collecting and providing more helpful information.

Our CUEZILLA prototype is such a tool and measures the quality of new bug reports; it also recommends which elements should be added to improve the quality. We trained CUEZILLA on a sample of 289 bug reports, rated by developers as part of the survey. In our experiments, CUEZILLA was able to predict the quality of 31--48% of bug reports accurately.

REFERENCES

• 1: G. Antoniol, H. Gall, M. D. Penta, and M. Pinzger. Mozilla: Closing the circle. Technical Report TUV-1841-2004-05, Technical University of Vienna, 2004.
• 2: John Anvik , Lyndon Hiew , Gail C. Murphy, Who should fix this bug?, Proceedings of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134336]
• 3: Victor R. Basili , Forrest Shull , Filippo Lanubile, Building Knowledge through Families of Experiments, IEEE Transactions on Software Engineering, v.25 n.4, p.456-473, July 1999 [doi>10.1109/32.799939]
• 4: Nicolas Bettenburg , Sascha Just , Adrian Schröter , Cathrin Weiß , Rahul Premraj , Thomas Zimmermann, Quality of bug reports in Eclipse, Proceedings of the 2007 OOPSLA workshop on eclipse technology eXchange, p.21-25, October 21-21, 2007, Montreal, Quebec, Canada [doi>10.1145/1328279.1328284]
• 5: N. Bettenburg, S. Just, A. Schröter, C. Weiss, R. Premraj, and T. Zimmermann. What makes a good bug report? Version 1.1. Technical report, Saarland University, Software Engineering Chair, March 2008. The technical report is an extended version of this paper. http://www.st.cs.uni-sb.de/publications/details/bettenburg-tr-2008/.
• 6: N. Bettenburg, R. Premraj, T. Zimmermann, and S. Kim. Duplicate bug reports considered harmful... really? In ICSM '08: Proceedings of the 24th IEEE International Conference on Software Maintenance, September 2008. To appear.
• 7: Nicolas Bettenburg , Rahul Premraj , Thomas Zimmermann , Sunghun Kim, Extracting structural information from bug reports, Proceedings of the 2008 international working conference on Mining software repositories, May 10-11, 2008, Leipzig, Germany [doi>10.1145/1370750.1370757]
• 8: Gerardo Canfora , Luigi Cerulo, Fine grained indexing of software repositories to support impact analysis, Proceedings of the 2006 international workshop on Mining software repositories, May 22-23, 2006, Shanghai, China [doi>10.1145/1137983.1138009]
• 9: Gerardo Canfora , Luigi Cerulo, Supporting change request assignment in open source development, Proceedings of the 2006 ACM symposium on Applied computing, April 23-27, 2006, Dijon, France [doi>10.1145/1141277.1141693]
• 10: L. Cherry and W. Vesterman. Writing tools - the STYLE and DICTION programs. Technical report, AT&T Laboratories, 1980.
• 11: D. Cubranic and G. C. Murphy. Automatic bug triage using text categorization. In SEKE 2004: Proceedings of the Sixteenth International Conference on Software Engineering & Knowledge Engineering, pages 92--97, 2004.
• 12: Michael Fischer , Martin Pinzger , Harald Gall, Analyzing and Relating Bug Report Data for Feature Tracking, Proceedings of the 10th Working Conference on Reverse Engineering, p.90, November 13-17, 2003
• 13: E. Goldberg. Bug writing guidelines. https://bugs.eclipse.org/bugs/bugwritinghelp.html. Last accessed 2007-08-04.
• 14: Pieter Hooimeijer , Westley Weimer, Modeling bug report quality, Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering, November 05-09, 2007, Atlanta, Georgia, USA [doi>10.1145/1321631.1321639]
• 15: HOT or NOT. http://www.hotornot.com/. Last accessed 2007-09-11.
• 16: S. Joshi and A. Orso. SCARPE: A Technique and Tool for Selective Record and Replay of Program Executions. In Proceedings of the 23rd IEEE International Conference on Software Maintenance (ICSM 2007), Paris, France, October 2007.
• 17: S. Just, R. Premraj, and T. Zimmermann. Towards the next generation of bug tracking systems. In VL/HCC '08: Proceedings of the 2008 IEEE Symposium on Visual Languages and Human-Centric Computing, September 2008. To appear.
• 18: Mik Kersten , Gail C. Murphy, Using task context to improve programmer productivity, Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering, November 05-11, 2006, Portland, Oregon, USA [doi>10.1145/1181775.1181777]
• 19: J. P. Kincaid, R. P. Fishburne, Jr., R. L. Rogers, and B. S. Chissom. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Research Branch Report 8--75, Millington, TN: Naval Technical Training, U. S. Naval Air Station, Memphis, TN, 1975.
• 20: Andrew J. Ko , Brad A. Myers , Duen Horng Chau, A Linguistic Analysis of How People Describe Software Problems, Proceedings of the Visual Languages and Human-Centric Computing, p.127-134, September 04-08, 2006 [doi>10.1109/VLHCC.2006.3]
• 21: Ben Liblit , Mayur Naik , Alice X. Zheng , Alex Aiken , Michael I. Jordan, Scalable statistical bug isolation, Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, June 12-15, 2005, Chicago, IL, USA
• 22: R. Likert. A technique for the measurement of attitudes. Archives of Psychology, 140:1--55, 1932.
• 23: D. MacKenzie, P. Eggert, and R. Stallman. Comparing and Merging Files with GNU Diff and Patch. Network Theory Ltd., 2003.
• 24: Leon Moonen, Generating Robust Parsers using Island Grammars, Proceedings of the Eighth Working Conference on Reverse Engineering (WCRE'01), p.13, October 02-05, 2001
• 25: A. Orso, S. Joshi, M. Burger, and A. Zeller. Isolating relevant component interactions with JINSI. In Proc. of Fifth International Workshop on Dynamic Analysis (WODA 2007), May 2006.
• 26: Lucas D. Panjer, Predicting Eclipse Bug Lifetimes, Proceedings of the Fourth International Workshop on Mining Software Repositories, p.29, May 20-26, 2007 [doi>10.1109/MSR.2007.25]
• 27: Ursula Passing , Martin Shepperd, An experiment on software project size and effort estimation, Proceedings of the 2003 International Symposium on Empirical Software Engineering, p.120, September 30-October 01, 2003
• 28: Teade Punter , Marcus Ciolkowski , Bernd Freimut , Isabel John, Conducting On-line Surveys in Software Engineering, Proceedings of the 2003 International Symposium on Empirical Software Engineering, p.80, September 30-October 01, 2003
• 29: Ratemyface.com. http://www.ratemyface.com/. Last accessed 2007-09-11.
• 30: Per Runeson , Magnus Alexandersson , Oskar Nyholm, Detection of Duplicate Defect Reports Using Natural Language Processing, Proceedings of the 29th international conference on Software Engineering, p.499-510, May 20-26, 2007 [doi>10.1109/ICSE.2007.32]
• 31: S. Siegel and N. J. Castellan, Jr. Nonparametric Statistics for the Behavioral Sciences. McGraw-Hill, second edition, 1988.
• 32: Janice Singer , Norman G. Vinson, Ethical Issues in Empirical Studies of Software Engineering, IEEE Transactions on Software Engineering, v.28 n.12, p.1171-1180, December 2002 [doi>10.1109/TSE.2002.1158289]
• 33: Joel Spolsky, Joel on Software: Selected Essays, APress, 2004
• 34: Xiaoyin Wang , Lu Zhang , Tao Xie , John Anvik , Jiasu Sun, An approach to detecting duplicate bug reports using natural language and execution information, Proceedings of the 30th international conference on Software engineering, May 10-18, 2008, Leipzig, Germany [doi>10.1145/1368088.1368151]
• 35: L. Wasserman. All of Statistics: A Concise Course in Statistical Inference. Springer, second edition, 2004.
• 36: Westley Weimer, Patches as better bug reports, Proceedings of the 5th international conference on Generative programming and component engineering, October 22-26, 2006, Portland, Oregon, USA [doi>10.1145/1173706.1173734]
• 37: Cathrin Weiss , Rahul Premraj , Thomas Zimmermann , Andreas Zeller, How Long Will It Take to Fix This Bug?, Proceedings of the Fourth International Workshop on Mining Software Repositories, p.1, May 20-26, 2007 [doi>10.1109/MSR.2007.13]
• 38: Guoqing Xu , Atanas Rountev , Yan Tang , Feng Qin, Efficient checkpointing of java software using context-sensitive capture and replay, Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, September 03-07, 2007, Dubrovnik, Croatia [doi>10.1145/1287624.1287638]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.7 Distribution, Maintenance, and Enhancement

General Terms:
Human Factors, Management, Measurement