Supporting cognitive walkthrough with video data: a mobile learning evaluation study

ABSTRACT

Although expert-based evaluation techniques such as heuristic evaluation and cognitive walkthrough are often inexpensive and quick to apply, they have not proved to be effective in capturing contextual factors that arise in real-world settings. It is no trivial issue to understand how such evaluation techniques could be modified or differently applied so as to better take into account context, without loosing the advantages inherent in those techniques. This paper explores a possible way of addressing the trade-off between application of cognitive walkthrough and low cost improvements of its contextual validity. In particular, we propose and investigate the benefits of supporting cognitive walkthrough with video data about user interaction with an eLearning course on mobile device. Initial results from this study indicated that video data provided evaluators with a more detailed understanding of user characteristics and interaction contexts, leading to an improvement of their assessments in terms of the total number of system's flaws detected. Video data was regarded by experts as both relevant and useful, especially for tuning the evaluation focus on types of difficulties they would normally not have experienced because of differences in terms of abilities, knowledge and background with those of the target user group.

REFERENCES

• 1: Gregory D. Abowd , Elizabeth D. Mynatt, Charting past, present, and future research in ubiquitous computing, ACM Transactions on Computer-Human Interaction (TOCHI), v.7 n.1, p.29-58, March 2000 [doi>10.1145/344949.344988]
• 2: Benyon, D., Turner, P. and Turner, S. Designing Interactive Systems . London: Addison Wesley, 2005.
• 3: Buisson, M. and Jestin, Y. In M D Dunlop and S A Brewster (eds): Proceedings of Mobile HCI 2001: 3rd Int. Workshop on IHM-HCI 2001. Design issues in distributed interaction supporting tools: mobile devices in an ATC Working Position., Lille, France, Sept. 2001, http://www.tls.cena.fr/divisions/PII/Rapports/NR01-764.pdf
• 4: Gilbert Cockton , Darryn Lavery , Alan Woolrych, Inspection-based evaluations, The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 2002
• 5: Dimitrova, M., Sharp H. and Wilson, S.: Are Experts Able to Predict Learner Problems during Usability Evaluations? World Conference on. Educational Multimedia, Hypermedia and Telecommunications, 2001(1), 419--424.
• 6: Dimitrova, M., Sharp H. and Wilson, S. Educational Multimedia Cognitive Walkthrough: Supporting Experts to Predict Valid User Problems. In Proceedings of Human Computer Interaction 2002, September 2-6, London, vol. 2 2002, 26--29.
• 7: Johnson, P. Usability and mobility; interaction on the move. Proc. First Workshop on Human-Computer Interaction with Mobile Devices, Glasgow, UK, 1998.
• 8: Kjeldskov, J. and Graham, C. A review of MobileHCI research methods. Proc. Mobile HCI 2003, Udine, Italy, 2003.
• 9: Wendy E. Mackay , Anne V. Ratzer , Paul Janecek, Video artifacts for design: bridging the Gap between abstraction and detail, Proceedings of the conference on Designing interactive systems: processes, practices, methods, and techniques, p.72-82, August 17-19, 2000, New York City, New York, United States [doi>10.1145/347642.347666]
• 10: Magoulas, G. D., Chen, S. Y., and Papanikolaou , K. A: Integrating Layered and Heuristic Evaluation for Adaptive Learning Environments. In Proc. of the 2nd Workshop on Empirical Evaluation of Adaptive Systems, 9th International Conference on User Modeling, 2003, University of Pittsburgh, 2003.
• 11: Jakob Nielsen, Heuristic evaluation, Usability inspection methods, John Wiley & Sons, Inc., New York, NY, 1994
• 12: David G. Novick, Using the cognitive walkthrough for operating procedures, interactions, v.6 n.3, p.31-37, May/June 1999 [doi>10.1145/301153.301166]
• 13: Po, S., Howard, S., Vetere, F., Skov M.B.. In S. Brewster and M. Dunlop (eds.) Proc. of Mobile HCI 2004. Heuristic Evaluation and Mobile Usability: Bridging the Realism Gap, 2004, 49--60.
• 14: Spencer, R.: The streamlined cognitive walkthrough method, working around social constraints encountered in a software development company. CHI Letters, 2(1), 2000, 353--359.
• 15: Lucy A. Suchman, Plans and situated actions: the problem of human-machine communication, Cambridge University Press, New York, NY, 1987
• 16: Cathleen Wharton , John Rieman , Clayton Lewis , Peter Polson, The cognitive walkthrough method: a practitioner's guide, Usability inspection methods, John Wiley & Sons, Inc., New York, NY, 1994

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

General Terms:
Human Factors

Keywords:
cognitive walkthrough, expert-based usability methods, mobile learning evaluation