Program comprehension as fact finding

ABSTRACT

Little is known about how developers think about design during code modification tasks or how experienced developers' design knowledge helps them work more effectively. We performed a lab study in which thirteen developers worked for 3 hours under-standing the design of a 54 KLOC open source application. Par-ticipants had from 0 to 10.5 years of industry experience and were grouped into three "experts" and ten "novices." We observed that participants spent their time seeking, learning, critiquing, explain-ing, proposing, and implementing facts about the code such as "getFoldLevel has effects". These facts served numerous roles, such as suggesting changes, constraining changes, and predicting the amount of additional investigation necessary to make a change. Differences between experts and novices included that the experts explained the root cause of the design problem and made changes to address it, while novice changes addressed only the symptoms. Experts did not read more methods but also did not visit some methods novices wasted time understanding. Experts talked about code in terms of abstractions such as "caching" while novices more often described code statement by statement. Ex-perts were able to implement a change faster than novices. Experts perceived problems novices did not and were able to explain facts novices could not. These findings have interesting implications for future tools.

REFERENCES

• 1: L. M. Berlin. Beyond Program Understanding: A Look at Programming Expertise in Industry. Empirical Studies of Programmers: Fifth Workshop, 6--25, 1993.
• 2: Paul Clements , David Garlan , Len Bass , Judith Stafford , Robert Nord , James Ivers , Reed Little, Documenting Software Architectures: Views and Beyond, Pearson Education, 2002
• 3: W. G. Chase and H. A. Simon. Perception in chess. In Cognitive Psychology, 1973.
• 4: S. P. Davies. Externalizing information during coding activities: Effects of expertise, environment and task. In Empirical Studies of Programmers: Fifth Workshop, 42--61, 1993.
• 5: F. Détienne, Program understanding and knowledge organization: the influence of acquired schemata, Cognitive ergonomics: understanding, learning and designing human-computer interaction, Academic Press Professional, Inc., San Diego, CA, 1990
• 6: Françoise Détienne , Frank Bott, Software design---cognitive aspects, Springer-Verlag New York, Inc., New York, NY, 2001
• 7: Erich Gamma , Richard Helm , Ralph Johnson , John Vlissides, Design patterns: elements of reusable object-oriented software, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1995
• 8: Leo Gugerty , Gary M. Olson, Comprehension differences in debugging by skilled and novice programmers, Papers presented at the first workshop on empirical studies of programmers on Empirical studies of programmers, p.13-27, June 1986, Washington, D.C., United States
• 9: Barbara A. Kitchenham , Tore Dyba , Magne Jorgensen, Evidence-Based Software Engineering, Proceedings of the 26th International Conference on Software Engineering, p.273-281, May 23-28, 2004
• 10: Andrew J. Ko , Robert DeLine , Gina Venolia, Information Needs in Collocated Software Development Teams, Proceedings of the 29th international conference on Software Engineering, p.344-353, May 20-26, 2007 [doi>10.1109/ICSE.2007.45]
• 11: A. J. Ko and B. A. Myers. A Framework and Methodology for Studying the Causes of Software Errors in Programming Systems. In Journal of Visual Languages and Computing, 16, 1-2, 41--84, 2005.
• 12: Andrew J. Ko , Brad A. Myers , Michael J. Coblenz , Htet Htet Aung, An Exploratory Study of How Developers Seek, Relate, and Collect Relevant Information during Software Maintenance Tasks, IEEE Transactions on Software Engineering, v.32 n.12, p.971-987, December 2006 [doi>10.1109/TSE.2006.116]
• 13: Thomas D. LaToza , Gina Venolia , Robert DeLine, Maintaining mental models: a study of developer work habits, Proceedings of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134355]
• 14: John M. Carroll , Thomas P. Moran, Design Rationale: Concepts, Techniques, and Use, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1996
• 15: Martin P. Robillard , Wesley Coelho , Gail C. Murphy, How Effective Developers Investigate Source Code: An Exploratory Study, IEEE Transactions on Software Engineering, v.30 n.12, p.889-903, December 2004 [doi>10.1109/TSE.2004.101]
• 16: D. L. Parnas, On the criteria to be used in decomposing systems into modules, Communications of the ACM, v.15 n.12, p.1053-1058, Dec. 1972 [doi>10.1145/361598.361623]
• 17: Dewayne E. Perry , Alexander L. Wolf, Foundations for the study of software architecture, ACM SIGSOFT Software Engineering Notes, v.17 n.4, p.40-52, Oct. 1992 [doi>10.1145/141874.141884]
• 18: Kevin J. Sullivan , William G. Griswold , Yuanfang Cai , Ben Hallen, The structure and value of modularity in software design, Proceedings of the 8th European software engineering conference held jointly with 9th ACM SIGSOFT international symposium on Foundations of software engineering, September 10-14, 2001, Vienna, Austria
• 19: Mary Shaw , David Garlan, Software architecture: perspectives on an emerging discipline, Prentice-Hall, Inc., Upper Saddle River, NJ, 1996
• 20: Thomas Zimmermann , Peter Weissgerber , Stephan Diehl , Andreas Zeller, Mining Version Histories to Guide Software Changes, IEEE Transactions on Software Engineering, v.31 n.6, p.429-445, June 2005 [doi>10.1109/TSE.2005.72]
• 21: A. Marie Vans , Anneliese von Mayhauser , Gabriel Somlo, Program understanding behavior during corrective maintenance of large-scale software, International Journal of Human-Computer Studies, v.51 n.1, p.31-70, July 1999 [doi>10.1006/ijhc.1999.0268]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.6 Programming Environments
          Subjects: Integrated environments

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.7 Distribution, Maintenance, and Enhancement
          Subjects: Restructuring, reverse engineering, and reengineering

General Terms:
Design, Human Factors

Keywords:
code navigation, empirical study, expertise, program comprehension, reverse engineering, science of design