Creating a cognitive metric of programming task difficulty

ABSTRACT

Conducting controlled experiments about programming activities often requires the use of multiple tasks of similar difficulty. In previously reported work about a controlled experiment investigating software exploration tools, we tried to select two change tasks of equivalent difficulty to be performed on a medium-sized code base. Despite careful effort in the selection and confirmation from our pilot subjects finding the two tasks to be of equivalent difficulty, the data from the experiment suggest the subjects found one of the tasks more difficult than the other.

In this paper, we report on early work to create a metric to estimate the cognitive difficulty for a software change task. Such a metric would help in comparing between studies of different tools, and in designing future studies. Our particular approach uses a graph-theoretic statistic to measure the complexity of the task solution by the connectedness of the solution elements. The metric predicts the perceived difficulty for the tasks of our experiment, but fails to predict the perceived difficulty for other tasks to a small program. We discuss these differences and suggest future approaches.

REFERENCES

• 1: B. Curtis , S. B. Sheppard , P. Milliman , M. A. Borst , T. Love, Measuring the Psychological Complexity of Software Maintenance Tasks with the Halstead and McCabe Metrics, IEEE Transactions on Software Engineering, v.5 n.2, p.96-104, March 1979 [doi>10.1109/TSE.1979.234165]
• 2: M. Czerwinski, E. Horvitz, and E. Cutrell. Subjective duration assessment: An implicit probe for software usability. In Proc. Joint IHM-HCI Conference, volume 2, pages 167--170, 2001.
• 3: Brian de Alwis , Gail C. Murphy , Martin P. Robillard, A Comparative Study of Three Program Exploration Tools, Proceedings of the 15th IEEE International Conference on Program Comprehension, p.103-112, June 26-29, 2007 [doi>10.1109/ICPC.2007.6]
• 4: Erich Gamma , Richard Helm , Ralph Johnson , John Vlissides, Design patterns: elements of reusable object-oriented software, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1995
• 5: Nicolas E. Gold , Andrew M. Mohan , Paul J. Layzell, Spatial Complexity Metrics: An Investigation of Utility, IEEE Transactions on Software Engineering, v.31 n.3, p.203-212, March 2005 [doi>10.1109/TSE.2005.39]
• 6: S. G. Hart and L. E. Staveland. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In P. A. Hancock and N. Meshkati, editors, Human Mental Workload, volume 52 of Advances in Psychology, pages 139--183. North-Holland, 1988.
• 7: Andrew J. Ko , Htet Aung , Brad A. Myers, Eliciting design requirements for maintenance-oriented IDEs: a detailed study of corrective and perfective maintenance tasks, Proceedings of the 27th international conference on Software engineering, May 15-21, 2005, St. Louis, MO, USA [doi>10.1145/1062455.1062492]
• 8: Katherine Isbister , Kristina Höök, Evaluating affective interfaces: innovative approaches, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1057115]
• 9: Martin P. Robillard , Wesley Coelho , Gail C. Murphy, How Effective Developers Investigate Source Code: An Exploratory Study, IEEE Transactions on Software Engineering, v.30 n.12, p.889-903, December 2004 [doi>10.1109/TSE.2004.101]
• 10: Mary Beth Rosson, Human factors in programming and software development, ACM Computing Surveys (CSUR), v.28 n.1, p.193-195, March 1996 [doi>10.1145/234313.234396]
• 11: Christos Tjortjis , Paul Layzell, Expert Maintainers' Strategies and Needs when Understanding Software: A Case Study Approach, Proceedings of the Eighth Asia-Pacific on Software Engineering Conference, p.281, December 04-07, 2001

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics

General Terms:
Algorithms, Measurement, Theory

Keywords:
cognitive metrics, experimentation, task difficulty