Evaluating usability based on multimodal information: an empirical study

ABSTRACT

New technologies are making it possible to provide an enriched view of interaction for researchers using multimodal information. This preliminary study explores the use of multiple information streams in usability evaluation. In the study, easy, medium and difficult versions of a game task were used to vary the levels of mental effort. Multimodal data streams during the three versions were analyzed, including eye tracking, pupil size, hand movement, heart rate variability (HRV) and subjectively reported data. Four findings indicate the potential value of usability evaluations based on multimodal information: First, subjective and physiological measures showed significant sensitivity to task difficulty. Second, different mental workload levels appeared to correlate with eye movement patterns, especially with a combined eye-hand movement measure. Third, HRV showed correlations with saccade speed. Finally, we present a new method using the ratio of eye fixations over mouse clicks to evaluate performance in more detail. These results warrant further investigations and take an initial step toward establishing usability evaluation methods based on multimodal information.

REFERENCES

• 1: Hollnagel, E. From human factors to cognitive systems engineering: Human-machine interactions in the 21st century. In Kitamura, M., and Kimura, I. (eds.), Anzen-no-Tankyu (Researches on Safety). ERCPublishing, Tokyo, 2003.
• 2: Andreassi, J. L. Psychophysiology: Human Behavior and Physiological Response, 4th Edition. Lawrence Erlbaum Associates, Mahwah, NJ, 2000.
• 3: Rosalind W. Picard, Affective computing, MIT Press, Cambridge, MA, 1997
• 4: Kramer, A. F. Physiological metrics of mental workload: A review of recent progress. In Damos, D. L. (ed.), Multiple-Task Performance, Taylor and Francis, London, 1991, 279--328.
• 5: Dennis W. Rowe , John Sibert , Don Irwin, Heart rate variability: indicator of user state as an aid to human-computer interaction, Proceedings of the SIGCHI conference on Human factors in computing systems, p.480-487, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274709]
• 6: Koji Takahashi , Minoru Nakayama , Yasutaka Shimizu, The response of eye-movement and pupil size to audio instruction while viewing a moving target, Proceedings of the 2000 symposium on Eye tracking research & applications, p.131-138, November 06-08, 2000, Palm Beach Gardens, Florida, United States [doi>10.1145/355017.355043]
• 7: Hopman, J. C. W., Kollee, L. A. A., Stoelinga, G. B. A., van Geijn, H. P. and van Ravenswaaij-Arts, C. M. A. Heart rate variability. Annals of Internal Medicine 1993, 118, 436--447.
• 8: Malik, M. Heart rate variability. Circulation 1996, 93, 1043--1065.
• 9: Pomeranz, B., and Macaulay, R. J. B., Caudill, M. A., et al. Assessment of autonomic function in humans by heart rate spectral analysis. Am J Physiol Heart Circ Physiol 1985, 248, 151--H153.
• 10: Pagani, M., Lombardi, F., Guzzetti, S., et al. Power spectral analysis of heart rate and arterial pressure variabilities as a maker of sympatho-vagal interaction in man and conscious dog. Cir Res 1986, 59, 178--93.
• 11: Ettema, J. H. and Ziclhuis, R. L. Physiological parameters of mental load. Ergonomics 1971, 14, 137--44.
• 12: Luczak, I. I. and Lauring, W. J. An analysis of heart rate variability. Ergonomics 1973, 16, 85--97.
• 13: Hyndman, B. W., and Gregory, J. R. Spectral analysis of sinus arrhythmia during mental loading. Ergonomics 1975, 18, 255--70.
• 14: Boutcher, S. H., Naugent, F. W., and Mclaren, P. F., et al. Heart period variability of trained and untrained men at rest and during mental challenge. Psychophysiology 1998, 35, 16--22.
• 15: Hyde, C., and Izard, C. E. Cardiac rhythmicities and attention in young children. Psychophysiology 1997, 34, 547--52.
• 16: Kim J. Vicente , D. Craig Thornton , Neville Moray, Spectral analysis of sinus arrhythmia: a measure of mental effort, Human Factors, v.29 n.2, p.171-182, April 1987
• 17: Tattersall, A. J., and Hockey, G. R. J. Level of operator control and changes in heart rate variability during simulated flight maintenance. Human Factors 1995, 37, 682--698.
• 18: Hess, E. H. and Polt, J. M. Pupil size in relation to mental activity during simple problem solving. Science 1964, 132, 1190--1192.
• 19: Hoecks, B., and Levelt, W. Pupillary dilation as a measure of attention: A quantitative system analysis. Behavior Research Methods, Instruments, & Computers 1993, 25, 16--26.
• 20: Juris, M. and Velden, M. The pupillary response to mental overload. Physiological Psychology 1977, 5 (4), 421--424.
• 21: Kahneman, D. Pupillary responses in a pitch-discrimination task. Perception & Psychophysics 1967, 2, 101--105.
• 22: Minoru Nakayama , Koji Takahashi , Yasutaka Shimizu, The act of task difficulty and eye-movement frequency for the 'Oculo-motor indices', Proceedings of the 2002 symposium on Eye tracking research & applications, March 25-27, 2002, New Orleans, Louisiana [doi>10.1145/507072.507080]
• 23: Beatty, J. Task-evoked pupillary responses, processing load and the structure of processing resources. Psychological Bulletin 1982, 91 (2), 276--292.
• 24: Shamsi T. Iqbal , Xianjun Sam Zheng , Brian P. Bailey, Task-evoked pupillary response to mental workload in human-computer interaction, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986094]
• 25: Hess, E. H., and Petrovich, S. B. Pupillary behavior in communication. In Siegman, A. W., and Feldstein, S. (eds.), Nonverbal Behavior and Communication. Erlbaum, Hillsdale, NJ, 1987, 327--348.
• 26: Rayner, K. Eye movements and information processing: 20 years of research. Psychological Bulletin 1998, 124 (3), 372--422.
• 27: Noton, D., and Stark, L. W. Scanpath in saccadic eye movements while viewing and recognizing patterns. Vision Research 1971a, 11, 929--942.
• 28: Goldberg, H., and Kotval, X. P. Computer interface evaluation using eye movements: Methods and constructs. International Journal of Industrial Ergonomics 1999, 24, 631--645.
• 29: Joseph H. Goldberg , Mark J. Stimson , Marion Lewenstein , Neil Scott , Anna M. Wichansky, Eye tracking in web search tasks: design implications, Proceedings of the 2002 symposium on Eye tracking research & applications, March 25-27, 2002, New Orleans, Louisiana [doi>10.1145/507072.507082]
• 30: Ellis, S., Candera, R., Misner, J., C. S., Lankford, C. P., and Hutshinson, T. E. Windows to the soul? What eye movements tell us about software usability? In Proceedings of the Usability Professionals' Association Conference, 1998, 151--178.
• 31: Cowen, L., Ball, L. J., and Delin, J. An eye movement analysis of web page usability. In Proceedings of the Annual HCI, 2002, 317--335.
• 32: Antti Aaltonen , Aulikki Hyrskykari , Kari-Jouko Räihä, 101 spots, or how do users read menus?, Proceedings of the SIGCHI conference on Human factors in computing systems, p.132-139, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274664]
• 33: Crosby, M. E., and Peterson, W. W. Using eye movements to classify search strategies. In Proceedings of the Human Factors Society 35th Annual Meeting. 1991, 1476--1480.
• 34: Andrews, T., and Coppola, D. Idiosyncratic characteristics of saccadic eye movements when viewing different visual environments. Vision Research 1999, 39, 2947--2953.
• 35: Y. Lin , W. J. Zhang , L. G. Watson, Using eye movement parameters for evaluating human-machine interface frameworks under normal control operation and fault detection situations, International Journal of Human-Computer Studies, v.59 n.6, p.837-873, December 2003 [doi>10.1016/S1071-5819(03)00122-8]
• 36: Wickens, C. D., and Holland, J. G., Engineering Psychology and Human Performance, 3rd Edition. Longman, London, 2000.
• 37: Hébert. a,b., Renée. a,b., Odrée Dionne-Fournelle. a., Martine Crê te. a. and S. J. Lupien. Physiological stress response to video game playing: The contribution of built-in music. Life Sciences 2005, 76, 2371--2380.
• 38: Jeffrey Rubin , Theresa Hudson, Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, John Wiley & Sons, Inc., New York, NY, 1994
• 39: Hart, S. G., and Staveland, L. E. Development of NASA-TLX (task load index): Results of experimental and theoretical research. In Hancock, P.A., and Meshakati, N. (eds.), Human Mental Workload. North-Holland, Amsterdam, 1988, 39--183.
• 40: Marshall, C., and Rossman, G. B. Designing Qualitative Research, 3rd Edition, Sage, Thousand Oaks, CA, 1999.
• 41: Wilson, G. M., and Sasse, M. A. Investigating the impact of audio degradations on users: Subjective vs. objective assessment methods. In Proceedings of OZCHI 2000: Interfacing Reality in the New Millennium, 2000, 135--142.
• 42: Gillian M. Wilson, Psychophysiological indicators of the impact of media quality on users, CHI '01 extended abstracts on Human factors in computing systems, March 31-April 05, 2001, Seattle, Washington [doi>10.1145/634067.634125]
• 43: Ekman, P., Levenson, R. W., and Friesen, W. V. Autonomic nervous system activity distinguishes among emotions. Science 1983, 221 (4616), 1208--1210.
• 44: Joseph F Hair, Jr. , Rolph E Anderson , Ronald L Tatham, Multivariate data analysis with readings (2nd ed.), Macmillan Publishing Co., Inc., Indianapolis, IN, 1986
• 45: Tao Lin , Masaki Omata , Wanhua Hu , Atsumi Imamiya, Do physiological data relate to traditional usability indexes?, Proceedings of the 19th conference of the computer-human interaction special interest group (CHISIG) of Australia on Computer-human interaction: citizens online: considerations for today and the future, November 21-25, 2005, Canberra, Australia
• 46: Michael Meehan , Brent Insko , Mary Whitton , Frederick P. Brooks, Jr., Physiological measures of presence in stressful virtual environments, Proceedings of the 29th annual conference on Computer graphics and interactive techniques, July 23-26, 2002, San Antonio, Texas
• 47: Lin, Y., Zhang, W. J. and Koubek, R. J. Effective Attention Allocation Behavior and its Measurement: A Preliminary Study, Interacting with Computers, 2004 Vol. 16, Issue 6, pp. 1195--1210.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

General Terms:
Experimentation, Human Factors

Keywords:
eye tracking, multimodal, physiological measures, usability