A survey of software learnability: metrics, methodologies and guidelines

ABSTRACT

It is well-accepted that learnability is an important aspect of usability, yet there is little agreement as to how learnability should be defined, measured, and evaluated. In this paper, we present a survey of the previous definitions, metrics, and evaluation methodologies which have been used for software learnability. Our survey of evaluation methodologies leads us to a new question-suggestion protocol, which, in a user study, was shown to expose a significantly higher number of learnability issues in comparison to a more traditional think-aloud protocol. Based on the issues identified in our study, we present a classification system of learnability issues, and demonstrate how these categories can lead to guidelines for addressing the associated challenges.

REFERENCES

• 1: Alain Abran , Adel Khelifi , Witold Suryn , Ahmed Seffah, Usability Meanings and Interpretations in ISO Standards, Software Quality Control, v.11 n.4, p.325-338, November 2003 [doi>10.1023/A:1025869312943]
• 2: Ron Baecker , Kellogg Booth , Sasha Jovicic , Joanna McGrenere , Gale Moore, Reducing the gap between what users know and what they need to know, Proceedings on the 2000 conference on Universal Usability, p.17-23, November 16-17, 2000, Arlington, Virginia, United States [doi>10.1145/355460.355467]
• 3: Bevan, N. and Macleod, M. (1994). Usability measurement in context. Behaviour and Information Technology. 13:132--145.
• 4: Keith A. Butler, Connecting theory and practice: a case study of achieving usability goals, Proceedings of the SIGCHI conference on Human factors in computing systems, p.85-88, April 1985, San Francisco, California, United States
• 5: John M. Carroll , Robert L. Mack , Clayton H. Lewis , Nancy L. Grischkowsky , Scott R. Robertson, Exploring exploring a word processor, Human-Computer Interaction, v.1 n.3, p.283-307, September 1985 [doi>10.1207/s15327051hci0103_3]
• 6: Carroll, J. M. (1990). The Nurnberg Funnel. MIT Press.
• 7: John M. Carroll , Caroline Carrithers, Training wheels in a user interface, Communications of the ACM, v.27 n.8, p.800-806, Aug 1984 [doi>10.1145/358198.358218]
• 8: John M. Carroll , Mary Beth Rosson, Paradox of the active user, Interfacing thought: cognitive aspects of human-computer interaction, MIT Press, Cambridge, MA, 1987
• 9: Davis, S. and Wiedenbeck, S. (1998). The effect of interaction style and training method on end user learning of software packages. Interacting with Computers. 11(2):147--172.
• 10: Alan Dix , Janet E. Finlay , Gregory D. Abowd , Russell Beale, Human-Computer Interaction (3rd Edition), Prentice-Hall, Inc., Upper Saddle River, NJ, 2003
• 11: Joseph F. Dumas , Janice C. Redish, A Practical Guide to Usability Testing, Greenwood Publishing Group Inc., Westport, CT, 1993
• 12: Elliott, G. J., Jones, E. and Barker, P. (2002). A grounded theory approach to modeling learnability of hypermedia authoring tools. Interacting with Computers. 14(5):547--574.
• 13: Ericsson, K. A. and Simon, H. A. (1984). Protocol Analysis: Verbal Reports as Data. MIT Press.
• 14: Marita Franzke, Turning research into practice: characteristics of display-based interaction, Proceedings of the SIGCHI conference on Human factors in computing systems, p.421-428, May 07-11, 1995, Denver, Colorado, United States [doi>10.1145/223904.223961]
• 15: Tom Gilb, Level 6: Why We Can't Get There from Here, IEEE Software, v.13 n.1, p.97-98, 103, January 1996
• 16: Tovi Grossman , Pierre Dragicevic , Ravin Balakrishnan, Strategies for accelerating on-line learning of hotkeys, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240865]
• 17: Kathy Haramundanis, Learnability in information design, Proceedings of the 19th annual international conference on Computer documentation, October 21-24, 2001, Sante Fe, New Mexico, USA [doi>10.1145/501516.501519]
• 18: Andreas Holzinger, Usability engineering methods for software developers, Communications of the ACM, v.48 n.1, p.71-74, January 2005 [doi>10.1145/1039539.1039541]
• 19: Andrew Howes , Richard M. Young, Predicting the learnability of task-action mappings, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.113-118, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108861]
• 20: Amy Hurst , Scott E. Hudson , Jennifer Mankoff, Dynamic detection of novice vs. skilled use without a task model, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240669]
• 21: Robin Jeffries , James R. Miller , Cathleen Wharton , Kathy Uyeda, User interface evaluation in the real world: a comparison of four techniques, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.119-124, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108862]
• 22: T. Kato, What “question-asking protocols” can say about the user interface, International Journal of Man-Machine Studies, v.25 n.6, p.659-673, Dec. 1986 [doi>10.1016/S0020-7373(86)80080-3]
• 23: Caitlin Kelleher , Randy Pausch, Stencils-based tutorials: design and evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems, April 02-07, 2005, Portland, Oregon, USA [doi>10.1145/1054972.1055047]
• 24: Kieras, D. E. and Polson, P. G. (1985). An approach to the formal analysis of user complexity. International Journal of Man-Machine Studies. 22:365--394.
• 25: Lazar, J., Jones, A. and Shneiderman, B. (2006). Workplace user frustration with computers: An exploratory investigation of the causes and severity. Behaviour and Info. Technology. 25(3):239--251.
• 26: J. C. R. Licklider, User-oriented interactive computer graphics, Proceedings of the ACM/SIGGRAPH workshop on User-oriented design of interactive graphics systems, October 14-15, 1976, Pittsburgh, PA [doi>10.1145/1024273.1024284]
• 27: Lin, H. X., Choong, Y. and Salvendy, G. (1997). A proposed index of usability: a method for comparing the relative usability of different software systems. Behaviour&Information Technology. 16:267--278.
• 28: Linja-aho, M. (2005). Evaluating and Improving the Learnability of a Building Modeling System. Helsinki University of Technology.
• 29: Robert Mack , Jill Burdett Robinson, When novices elicit knowledge: question asking in designing, evaluating, and learning to use software, The psychology of expertise: cognitive research and empirical AI, Springer-Verlag New York, Inc., Secaucus, NJ, 1992
• 30: Robert L. Mack , Clayton H. Lewis , John M. Carroll, Learning to use word processors: problems and prospects, ACM Transactions on Information Systems (TOIS), v.1 n.3, p.254-271, July 1983 [doi>10.1145/357436.357440]
• 31: Wendy E. Mackay, Triggers and barriers to customizing software, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.153-160, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108867]
• 32: Christie D. Michelsen , Wayne D. Dominick , Joseph E. Urban, A methodology for the objective evaluation of the user/system interfaces of the MADAM system using software engineering principles, Proceedings of the 18th annual Southeast regional conference, March 24-26, 1980, Tallahassee, Florida [doi>10.1145/503838.503847]
• 33: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1995
• 34: Susan Palmiter , Jay Elkerton, An evaluation of animated demonstrations of learning computer-based tasks, Proceedings of the SIGCHI conference on Human factors in computing systems: Reaching through technology, p.257-263, April 27-May 02, 1991, New Orleans, Louisiana, United States [doi>10.1145/108844.108906]
• 35: Tim F. Paymans , Jasper Lindenberg , Mark Neerincx, Usability trade-offs for adaptive user interfaces: ease of use and learnability, Proceedings of the 9th international conference on Intelligent user interfaces, January 13-16, 2004, Funchal, Madeira, Portugal [doi>10.1145/964442.964512]
• 36: John Rieman, A field study of exploratory learning strategies, ACM Transactions on Computer-Human Interaction (TOCHI), v.3 n.3, p.189-218, Sept. 1996 [doi>10.1145/234526.234527]
• 37: Paulo J. Santos , Albert N. Badre, Automatic chunk detection in human-computer interaction, Proceedings of the workshop on Advanced visual interfaces, p.69-77, June 01-04, 1994, Bari, Italy [doi>10.1145/192309.192327]
• 38: Santos, P. J. and Badre, A. N. (1995). Discount learnability evaluation. GVU Technical Report GIT-GVU-95-30. Georgia Institute of Technology.
• 39: B. Shneiderman, Direct Manipulation: A Step Beyond Programming Languages, Computer, v.16 n.8, p.57-69, August 1983 [doi>10.1109/MC.1983.1654471]
• 40: Ben Shneiderman, Designing the User Interface: Strategies for Effective Human-Computer Interaction, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1997
• 41: Stickel, C., Fink, J. and Holzinger, A. (2007). Enhancing Universal Access-EEG Based Learnability Assessment. Lecture Notes in Comp. Sci. 813--822.
• 42: Michael B. Twidale, Over the Shoulder Learning: Supporting Brief Informal Learning, Computer Supported Cooperative Work, v.14 n.6, p.505-547, December 2005 [doi>10.1007/s10606-005-9007-7]
• 43: Dennis Wixon, Evaluating usability methods: why the current literature fails the practitioner, interactions, v.10 n.4, July + August 2003 [doi>10.1145/838830.838870]
• 44: Jun Xiao , John Stasko , Richard Catrambone, An Empirical Study of the Effect of Agent Competence on User Performance and Perception, Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, p.178-185, July 19-23, 2004, New York, New York [doi>10.1109/AAMAS.2004.61]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

General Terms:
Experimentation, Human Factors

Keywords:
evaluation, learnability, learning, question-suggestion, software, think-aloud, usability