Combining self-reported and automatic data to improve programming effort measurement

ABSTRACT

Measuring effort accurately and consistently across subjects in a programming experiment can be a surprisingly difficult task. In particular, measures based on self-reported data may differ significantly from measures based on data which is recorded automatically from a subject's computing environment. Since self-reports can be unreliable, and not all activities can be captured automatically, a complete measure of programming effort should incorporate both classes of data. In this paper, we show how self-reported and automatic effort can be combined to perform validation and to measure total programming effort.

REFERENCES

• 1: David Atkins , Thomas Ball , Todd Graves , Audris Mockus, Using version control data to evaluate the impact of software tools, Proceedings of the 21st international conference on Software engineering, p.324-333, May 16-22, 1999, Los Angeles, California, United States [doi>10.1145/302405.302649]
• 2: Basili, V.R., and Reiter, R.J. Evaluating automatable measures of software development. In Proceedings of the Workshop on Quantitative Software Models for Reliability, Complexity and Cost (Oct. 1979).
• 3: Basili, V.R., Selby, W.R., and Phillips, T.-Y. Metric analysis and data validation Across Fortran projects. IEEE Transactions on Software Engineering, SE-9, 6 (Nov. 1983), 652--663.
• 4: Campbell, D.T., and Stanley, J.C. Experimental and Quasi-Experimental Designs for Research. Houghton-Mifflin, 1963.
• 5: Philip Johnson, Workshop on Software Engineering for High Performance Computing System (HPCS) Applications, Proceedings of the 26th International Conference on Software Engineering, p.772, May 23-28, 2004
• 6: Leonardo Dagum , Ramesh Menon, OpenMP: An Industry-Standard API for Shared-Memory Programming, IEEE Computational Science & Engineering, v.5 n.1, p.46-55, January 1998 [doi>10.1109/99.660313]
• 7: Jack J. Dongarra , Steve W. Otto , Marc Snir , David Walker, A message passing standard for MPP and workstations, Communications of the ACM, v.39 n.7, p.84-90, July 1996 [doi>10.1145/233977.234000]
• 8: Watts S. Humphrey, A Discipline for Software Engineering, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1995
• 9: Philip M. Johnson , Hongbing Kou , Joy Agustin , Christopher Chan , Carleton Moore , Jitender Miglani , Shenyan Zhen , William E. J. Doane, Beyond the Personal Software Process: metrics collection and analysis for the differently disciplined, Proceedings of the 25th International Conference on Software Engineering, May 03-10, 2003, Portland, Oregon
• 10: Kou, H., and Xu, X. Most Active File Measurement Validation In Hackystat. Technical Report CSDL-02-09, University of Hawaii, Honolulu, HI, 2002.
• 11: Perry, D.E., Staudenmayer, N.A., and Votta, L.G. Understanding and improving time usage in software development. In Volume 5 of Trends in Software: Software Process, John Wiley & Sons, 1995.
• 12: Szafron, D., and Schaeffer, J. An experiment to measure the usability of parallel programming systems. In Concurrency: Practice and Experience, 8, 2 (Mar. 1996), 147--166.
• 13: Thomas, R., Kennedy, G.E., Draper, S., Mancy, R., Crease, M., Evans, H., and Gray, P. Generic usage monitoring of programming students. In Proceedings of the 20th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE '03) (Adelaide, Australia, Dec 7-10, 2003).
• 14: Marvin Zelkowitz , Victor Basili , Sima Asgari , Lorin Hochstein , Jeff Hollingsworth , Taiga Nakamura, Measuring Productivity on High Performance Computers, Proceedings of the 11th IEEE International Software Metrics Symposium (METRICS'05), p.6, September 19-22, 2005 [doi>10.1109/METRICS.2005.33]

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Process metrics

General Terms:
Experimentation, Human Factors, Measurement, Verification

Keywords:
effort, manual approaches