Does time heal?: a longitudinal study of usability

ABSTRACT

We report from a longitudinal laboratory-based usability evaluation of an interactive system. A usability evaluation was conducted with novice users when a large commercial electronic patient record system was being deployed in the use organization. After the users had used the system in their daily work for 15 months, same evaluation was conducted again. Our aim was to inquire into the nature of usability problems experienced by novice and expert users over time, and to see to what extends usability problems may or may not disappear over time, as users get more familiar with the system. On the basis of our two usability evaluations, we present key findings on the usability of the evaluated system as experienced by the two categories of users at these two different points in time. Based on our findings, we discuss implications for evaluating usability.

REFERENCES

• 1: Bryce Allen, Cognitive abilities and information system usability, Information Processing and Management: an International Journal, v.30 n.2, p.177-191, March-April 1994 [doi>10.1016/0306-4573(94)90063-9]
• 2: Bailey, R. W., Allan, R. W., Riello, P. (1992). Usability Testing vs. Heuristic Evaluation: A Head-to-Head Comparison. In Proceedings of the Human Factors and Ergonomics Society 36th Annual Meeting, HFES, (409--413).
• 3: Roman Bednarik , Markku Tukiainen, Effects of display blurring on the behavior of novices and experts during program debugging, CHI '05 extended abstracts on Human factors in computing systems, April 02-07, 2005, Portland, OR, USA [doi>10.1145/1056808.1056877]
• 4: Bourie, P. Q., Dresch, J., and Chapman, R. H. (1997). Usability Evaluation of an On-line Nursing Assessment. In Proceedings of AMIA Symposium.
• 5: Dillon, A. and Song, M. (1997). An Empirical Comparison of the Usability for Novice and Expert Searchers of a Textual and a Graphic Interface to an Art-Resource Database. Journal of Digital Information, 1(1).
• 6: Alan Dix , Janet Finley , Gregory Abowd , Russell Beale, Human-computer interaction (2nd ed.), Prentice-Hall, Inc., Upper Saddle River, NJ, 1998
• 7: Frøkjær, E., Hertzum, M. and Hornbæk, K. (2000) Measuring Usability: Are Effectiveness, Efficiency, and Satisfaction Really Correlated? CHI Letters, 2(1). 345--352.
• 8: Hart, S. G., Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Hancock, P. A. and Meshkati, N. (Eds.), Human Mental Workload (pp. 139--183). Amsterdam: Elsevier Science Publishers.
• 9: Norio Ishii , Kazuhisa Miwa, Interactive processes between mental and external operations in creative activity: a comparison of experts' and novices' performance, Proceedings of the 4th conference on Creativity & cognition, p.178-185, October 13-16, 2002, Loughborough, UK [doi>10.1145/581710.581737]
• 10: ISO 9241 (1997). Ergonomic Requirements for Office Work with Visual Display Terminals. ISO.
• 11: Niels Ebbe Jacobsen , Morten Hertzum , Bonnie E. John, The evaluator effect in usability tests, CHI 98 conference summary on Human factors in computing systems, p.255-256, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/286498.286737]
• 12: Claire-Marie Karat , Robert Campbell , Tarra Fiegel, Comparison of empirical testing and walkthrough methods in user interface evaluation, Proceedings of the SIGCHI conference on Human factors in computing systems, p.397-404, May 03-07, 1992, Monterey, California, United States [doi>10.1145/142750.142873]
• 13: Wendy A. Kellogg , Timothy J. Breen, Evaluating user and system models: applying scaling techniques to problems in human-computer interaction, Proceedings of the SIGCHI/GI conference on Human factors in computing systems and graphics interface, p.303-308, April 05-09, 1987, Toronto, Ontario, Canada
• 14: Molich, R. (2000). Usable Web Design (In Danish). Ingeniøren|bøger.
• 15: NASA. Task Load Index, http://iac.dtic.mil/hsiac/Products.htm#TLX.
• 16: Nielsen, J. (2000). Novice vs. Expert Users. Alertbox, February 6, 2000. http://www.useit.com/alertbox/20000206.html
• 17: Jakob Nielsen, Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 18: Jennifer Preece , Jenny Preece , Yvonne Rogers , Helen Sharp, Beyond Interaction Design: Beyond Human-Computer Interaction, John Wiley & Sons, Inc., New York, NY, 2001
• 19: Jochen Prümper , Michael Frese , Dieter Zapf , Felix C. Brodbeck, Errors in computerized office work: differences between novice and expert users, ACM SIGCHI Bulletin, v.23 n.2, p.63-66, April 1991 [doi>10.1145/122488.122498]
• 20: Jeffrey Rubin , Theresa Hudson, Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, John Wiley & Sons, Inc., New York, NY, 1994
• 21: Haruhiko Urokohara , Kenichi Tanaka , Kazuyoshi Furuta , Michiyo Honda , Masaaki Kurosu, NEM: "novice expert ratio method" a usability evaluation method to generate a new performance measure, CHI '00 extended abstracts on Human factors in computing systems, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/633292.633394]

INDEX TERMS

Keywords:
electronic patient records, experts and novices, longitudinal, usability