How to compare program comprehension in FOSD empirically: an experience report

ABSTRACT

There are many different implementation approaches to realize the vision of feature-oriented software development, ranging from simple preprocessors, over feature-oriented programming, to sophisticated aspect-oriented mechanisms. Their impact on readability and maintainability (or program comprehension in general) has caused a debate among researchers, but sound empirical results are missing. We report experience from our endeavor to conduct experiments to measure the influence of different implementation mechanisms on program comprehension. We describe how to design such experiments and report from possibilities and pitfalls we encountered. Finally, we present some early results of our first experiment on comparing the CPP tool with the CIDE tool.

REFERENCES

• 1: J. R. Anderson. Cognitive Psychology and its Implications. Worth Publishers, 2000.
• 2: T. W. Anderson and J. D. Finn. The New Statistical Analysis of Data. Springer, 1996.
• 3: S. Apel et al. FeatureC++: On the Symbiosis of Feature-Oriented and Aspect-Oriented Programming. In Proc. Int'l Conf. Generative Programming and Component Engineering, pages 125--140, 2005.
• 4: S. Apel and C. Kästner. An Overview of Feature-Oriented Software Development. Journal of Object Technology (JOT), 8(5):49--84, 2009.
• 5: S. Apel, C. Kästner, and S. Trujillo. On the Necessity of Empirical Studies in the Assessment of Modularization Mechanisms for Crosscutting Concerns. In Proc. Int'l Workshop on Assessment of Contemporary Modularization Techniques, pages 1--7. 2007.
• 6: I. Aracic, V. Gasiunas, M. Mezini, and K. Ostermann. An Overview of CaesarJ. Transactions on Aspect-Oriented Software Development 1, pages 135--173, 2006.
• 7: A. P. Association. Publication Manual of the American Psychological Association. American Psychological Association, 2001.
• 8: D. Batory, J. N. Sarvela, and A. Rauschmayer. Scaling Step-Wise Refinement. IEEE Trans. Softw. Eng., 30(6):355--371, 2004.
• 9: R. E. Brooks. Using a Behavioral Theory of Program Comprehension in Software Engineering. In Proc. Int'l Conf. Software Engineering, pages 196--201. 1978.
• 10: J. Daly et al. The Effect of Inheritance on the Maintainability of Object-Oriented Software: An Empirical Study. In Proc. Int'l Conf. Software Maintenance, pages 20--29. 1995.
• 11: A. Dunsmore and M. Roper. A Comparative Evaluation of Program Comprehension Measures. Technical Report EFoCS-35-2000, University of Strathclyde, 2000.
• 12: S. Easterbrook, J. Singer, M.-A. Storey, and D. Damian. Guide to Advanced Empirical Software Engineering, pages 285--311. Springer, 2008.
• 13: J. Favre. Understanding-In-The-Large. In Proc. Int'l Workshop on Program Comprehension, page 29. 1997.
• 14: J. Feigenspan. Empirical Comparison of FOSD Approaches Regarding Program Comprehension -- A Feasibility Study. Master's thesis, University of Magdeburg, 2009.
• 15: E. Figueiredo et al. Evolving Software Product Lines with Aspects: An Empirical Study on Design Stability. In Proc. Int'l Conf. Software Engineering, pages 261--270. 2008.
• 16: C. J. Goodwin. Research In Psychology: Methods and Design. Wiley Publishing, Inc., 1998.
• 17: S. Hanenberg, S. Kleinschmager, and M. Josupeit-Walter. Does Aspect-Oriented Programming Increase The Development Speed for Crosscutting Code? An Empirical Study. In Proc. Int'l Symp. Empirical Software Engineering and Measurement, 2009.
• 18: S. Henry, M. Humphrey, and J. Lewis. Evaluation of the Maintainability of Object-Oriented Software. In Proc. TENCON, pages 404--409. 1990.
• 19: A. Höfer and W. Tichy. Empirical Software Engineering Issues. Critical Assessment and Future Directions, chapter Status of Empirical Research in Software Engineering, pages 10--19. Springer, 2007.
• 20: International Organization for Standardization. ISO/IEC 9899-1999: Programming Languages---C, 1999.
• 21: S. Jarzabek, P. Bassett, H. Zhang, and W. Zhang. XVCL: XML-based Variant Configuration Language. In Proc. Int'l Conf. Software Engineering, pages 810--811. 2003.
• 22: N. Juristo and A. M. Moreno. Basics of Software Engineering Experimentation. Kluwer, 2001.
• 23: V. B. Kampenes et al. A Systematic Review of Quasi-Experiments in Software Engineering. Information and Software Technology, 51(1):71--82, 2009.
• 24: C. Kästner, S. Apel, and M. Kuhlemann. Granularity in Software Product Lines. In Proc. Int'l Conf. Software Engineering, pages 311--320. 2008.
• 25: C. Kästner, S. Trujillo, and S. Apel. Visualizing Software Product Line Variabilities in Source Code. In Proc. Workshop Visualization in Software Product Line Engineering, 2008.
• 26: G. Kiczales et al. Aspect-Oriented Programming. In Proc. Europ. Conf. Object-Oriented Programming, pages 220--242. 1997.
• 27: G. Kiczales et al. An Overview of AspectJ. In Proc. Europ. Conf. Object-Oriented Programming, pages 327--353. 2001.
• 28: J. Koenemann and S. P. Robertson. Expert Problem Solving Strategies for Program Comprehension. In Proc. Conf. Human Factors in Computing Systems, pages 125--130. 1991.
• 29: B. P. Lientz and E. B. Swanson. Software Maintenance Management. Addison-Wesley, 1980.
• 30: R. Lopez-Herrejon, D. Batory, and W. Cook. Evaluating Support for Features in Advanced Modularization Technologies. In Proc. Europ. Conf. Object-Oriented Programming, pages 169--194. 2005.
• 31: M. Mezini and K. Ostermann. Variability Management with Feature-Oriented Programming and Aspects. In Proc. Int'l Symp. Foundations of Software Engineering, pages 127--136. 2004.
• 32: S. Patig. A Practical Guide to Testing the Understandability of Notations. In Proc. Asia-Pacific Conf. Conceptual Modelling, pages 49--58, 2008.
• 33: N. Pennington. Stimulus Structures and Mental Representations in Expert Comprehension of Computer Programs. Cognitive Psychologys, 19(3):295--341, 1987.
• 34: K. Popper. The Logic of Scientic Discovery. Routledge, 1959.
• 35: L. Prechelt et al. Two Controlled Experiments Assessing the Usefulness of Design Pattern Documentation in Program Maintenance. IEEE Trans. Softw. Eng., 28(6):595--606, 2002.
• 36: C. Prehofer. Feature-Oriented Programming: A Fresh Look at Objects. In Proc. Europ. Conf. Object-Oriented Programming, pages 419--443. 1997.
• 37: F. J. Roethlisberger. Management and the Worker. Harvard University Press, 1939.
• 38: R. Rosenthal and L. Jacobson. Teachers' Expectancies: Determinants of Pupils' IQ Gains. Psychological Reports, 19(1):115--118, 1966.
• 39: W. R. Shadish, T. D. Cook, and D. T. Campbell. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin, 2002.
• 40: D. I. K. Sjoberg et al. A Survey of Controlled Experiments in Software Engineering. IEEE Trans. Softw. Eng., 31(9):733--753, 2005.
• 41: H. Spencer and G. Collyer. #ifdef Considered Harmful or Portability Experience With C News. In Proc. USENIX Conf., pages 185--198, 1992.
• 42: A. von Mayrhauser and A. M. Vans. Program Comprehension During Software Maintenance and Evolution. Computer, 28(8):44--55, 1995.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.2 Design Tools and Techniques

Additional Classification:
  D. Software
  D.3 PROGRAMMING LANGUAGES
      D.3.3 Language Constructs and Features

General Terms:
Experimentation, Human Factors, Languages

Keywords:
CIDE, FOSD, empirical software engineering, preprocessors, program comprehension