Towards an index of opportunity: understanding changes in mental workload during task execution

ABSTRACT

To contribute to systems that reason about human attention, our work empirically demonstrates how a user's mental workload changes during task execution. We conducted a study where users performed interactive, hierarchical tasks while mental workload was measured through the use of pupil size. Results show that (i) different types of subtasks impose different mental workload, (ii) workload decreases at subtask boundaries, (iii) workload decreases more at boundaries higher in a task model and less at boundaries lower in the model, (iv) workload changes among subtask boundaries within the same level of a task model, and (v) effective understanding of why changes in workload occur requires that the measure be tightly coupled to a validated task model. From the results, we show how to map mental workload onto a computational Index of Opportunity that systems can use to better reason about human attention.

REFERENCES

• 1: Piotr D. Adamczyk , Brian P. Bailey, If not now, when?: the effects of interruption at different moments within task execution, Proceedings of the SIGCHI conference on Human factors in computing systems, p.271-278, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985727]
• 2: Bailey, B.P., J.A. Konstan and J.V. Carlis. The Effects of Interruptions on Task Performance, Annoyance, and Anxiety in the User Interface. INTERACT, 2001, 593--601.
• 3: Beatty, J. Task-Evoked Pupillary Responses, Processing Load, and the Structure of Processing Resources. Psychological Bulletin, 91 (2), 276--292, 1982.
• 4: Bradshaw, J.L. Pupil Size as a Measure of Arousal During Information Processing. Nature, 216, 515--516, 1967.
• 5: Stuart K. Card , Allen Newell , Thomas P. Moran, The Psychology of Human-Computer Interaction, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1983
• 6: Cutrell, E., M. Czerwinski and E. Horvitz. Notification, Disruption and Memory: Effects of Messaging Interruptions on Memory and Performance. INTERACT, 2001, 263--269.
• 7: Czerwinski, M., E. Cutrell and E. Horvitz. Instant Messaging and Interruption: Influence of Task Type on Performance. Proc. OZCHI, 2000, 356--361.
• 8: Czerwinski, M., E. Cutrell and E. Horvitz. Instant Messaging: Effects of Relevance and Timing. In People and Computers XIV: Proceedings of HCI, 2000, 71--76.
• 9: Gillie, T. and D. Broadbent. What Makes Interruptions Disruptive? A Study of Length, Similarity, and Complexity. Psychological Research, 50, 243--250, 1989.
• 10: Hess, E.H. and J.M. Polt. Pupil Size in Relation to Mental Activity During Simple Problem Solving. Science, 132, 1190--1192, 1964.
• 11: Hoecks, B. and W. Levelt. Pupillary Dilation as a Measure of Attention: A Quantitative System Analysis. Behavior Research Methods, Instruments, & Computers, 25, 16--26, 1993.
• 12: Eric Horvitz , Johnson Apacible, Learning and reasoning about interruption, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958440]
• 13: Horvitz, E., J. Breese, D. Heckerman, D. Hovel and K. Rommelse. The Lumiere Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users. Proc. Uncertainty in Artificial Intelligence, 1998, 256--265.
• 14: Horvitz, E., A. Jacobs and D. Hovel. Attention-Sensitive Alerting. Proc. Uncertainty in Artificial Intelligence, 1999, 305--313.
• 15: Scott Hudson , James Fogarty , Christopher Atkeson , Daniel Avrahami , Jodi Forlizzi , Sara Kiesler , Johnny Lee , Jie Yang, Predicting human interruptibility with sensors: a Wizard of Oz feasibility study, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642657]
• 16: Hytintk, J., J. Tommola and A. Alaja. Pupil Dilation as a Measure of Processing Load in Simultaneous Interpretation and Other Language Tasks. The Quarterly Journal of Experimental Psychology, 48A (3), 598--612, 1995.
• 17: Shamsi T. Iqbal , Xianjun Sam Zheng , Brian P. Bailey, Task-evoked pupillary response to mental workload in human-computer interaction, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986094]
• 18: Jackson, T.W., R.J. Dawson and D. Wilson. The Cost of Email Interruption. Journal of Systems and Information Technology, 5 (1), 81--92, 2001.
• 19: Bonnie John , Alonso Vera , Michael Matessa , Michael Freed , Roger Remington, Automating CPM-GOMS, Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/503376.503404]
• 20: Juris, M. and M. Velden. The Pupillary Response to Mental Overload. Physiological Psychology, 5 (4), 421--424, 1977.
• 21: Kahneman, D. Pupillary Responses in a Pitch-Discrimination Task. Perception & Psychophysics, 2, 101--105, 1967.
• 22: Kramer, A.F. Physiological Metrics of Mental Workload: A Review of Recent Progress. In Damos, D.L. ed. Multiple-Task Performance, Taylor and Francis, London, 1991, 279--328.
• 23: Kreifeldt, J.G. and M.E. McCarthy. Interruption as a Test of the User-Computer Interface. In Proceedings of the 17th Annual Conference on Manual Control, Jet Propulsion Laboratory, California Institute of Technology, JPL Publication 81-95, 1981, 655--667.
• 24: Pattie Maes, Agents that reduce work and information overload, Communications of the ACM, v.37 n.7, p.30-40, July 1994 [doi>10.1145/176789.176792]
• 25: Sandra Marshall, New techniques for evaluating innovative interfaces with eye tracking, Proceedings of the 5th international conference on Multimodal interfaces, p.2-2, November 05-07, 2003, Vancouver, British Columbia, Canada [doi>10.1145/958432.958434]
• 26: McFarlane, D.C. Coordinating the Interruption of People in Human-Computer Interaction. INTERACT, 1999, 295--303.
• 27: Miyata, Y. and D.A. Norman. The Control of Multiple Activities. In Norman, D.A. and Draper, S.W. (eds.) User Centered System Design: New Perspectives on Human-Computer Interaction, Lawrence Erlbaum Associates, Hillsdale, NJ, 1986.
• 28: Minoru Nakayama , Koji Takahashi , Yasutaka Shimizu, The act of task difficulty and eye-movement frequency for the 'Oculo-motor indices', Proceedings of the 2002 symposium on Eye tracking research & applications, March 25-27, 2002, New Orleans, Louisiana [doi>10.1145/507072.507080]
• 29: Rosalind W. Picard, Affective computing, MIT Press, Cambridge, MA, 1997
• 30: Dennis W. Rowe , John Sibert , Don Irwin, Heart rate variability: indicator of user state as an aid to human-computer interaction, Proceedings of the SIGCHI conference on Human factors in computing systems, p.480-487, April 18-23, 1998, Los Angeles, California, United States [doi>10.1145/274644.274709]
• 31: Jeffrey S. Shell , Ted Selker , Roel Vertegaal, Interacting with groups of computers, Communications of the ACM, v.46 n.3, March 2003 [doi>10.1145/636772.636796]
• 32: Speier, C., J.S. Valacich and I. Vessey. The Influence of Task Interruption on Individual Decision Making: An Information Overload Perspective. Decision Sciences, 30 (2), 337--360, 1999.
• 33: Koji Takahashi , Minoru Nakayama , Yasutaka Shimizu, The response of eye-movement and pupil size to audio instruction while viewing a moving target, Proceedings of the 2000 symposium on Eye tracking research & applications, p.131-138, November 06-08, 2000, Palm Beach Gardens, Florida, United States [doi>10.1145/355017.355043]
• 34: Tobii-Systems. http://www.tobii.se/
• 35: Gillian M. Wilson , M. Angela Sasse, The head or the heart?: measuring the impact of media quality, CHI '00 extended abstracts on Human factors in computing systems, April 01-06, 2000, The Hague, The Netherlands [doi>10.1145/633292.633360]
• 36: Zijlstra, F.R.H., R.A. Roe, A.B. Leonora and I. Krediet. Temporal Factors in Mental Work: Effects of Interrupted Activities. Journal of Occupational and Organizational Psychology, 72, 163--185, 1999.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: User-centered design

General Terms:
Design, Experimentation, Human Factors, Measurement

Keywords:
attention, interruption, pupil size, task models