Feasibility and pragmatics of classifying working memory load with an electroencephalograph

ABSTRACT

A reliable and unobtrusive measurement of working memory load could be used to evaluate the efficacy of interfaces and to provide real-time user-state information to adaptive systems. In this paper, we describe an experiment we con-ducted to explore some of the issues around using an elec-troencephalograph (EEG) for classifying working memory load. Within this experiment, we present our classification methodology, including a novel feature selection scheme that seems to alleviate the need for complex drift modeling and artifact rejection. We demonstrate classification accuracies of up to 99% for 2 memory load levels and up to 88% for 4 levels. We also present results suggesting that we can do this with shorter windows, much less training data, and a smaller number of EEG channels, than reported previously. Finally, we show results suggesting that the models we construct transfer across variants of the task, implying some level of generality. We believe these findings extend prior work and bring us a step closer to the use of such technologies in HCI research.

REFERENCES

• 1: Baddeley, A.D., & Hitch, G. (1974). Working Memory. In G.H Bower (ed.) Recent Advances in Learning and Motivation, 8. Academic Press: New York.
• 2: Berger, H. (1929). Uber das elektroenzephalogramm des menschen. Arch Psychiatr, 87, 527--570.
• 3: Stuart K. Card , Allen Newell , Thomas P. Moran, The Psychology of Human-Computer Interaction, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 1983
• 4: Daniel Chen , Roel Vertegaal, Using mental load for managing interruptions in physiologically attentive user interfaces, CHI '04 extended abstracts on Human factors in computing systems, April 24-29, 2004, Vienna, Austria [doi>10.1145/985921.986103]
• 5: Fisch, B.J. (2005). Fisch & Spehlmann's EEG primer: Basic principles of digital and analog EEG. Elsevier; Amsterdam.
• 6: Gevins, A., & Smith, M.E. (2003). Neurophysiological measures of cognitive workload during human-computer interaction. Theoretical Issues in Ergonomics Science, 4, 113--131.
• 7: Gevins, A., Smith, M.E., Leong, H., & McEvoy, L. (1998). Monitoring working memory load during computer-based tasks with EEG pattern recognition methods. Human Factors, 40(1), 79--91.
• 8: Howard, M.W., Rizzuto, D.S., Caplan, J.B., Madsen, J.R., Lisman, J., Aschenbrenner-Scheibe, R., Schulze-Bonhage, A., & Kahana, M.J. (2003). Gamma oscillations correlate with working memory load in humans. Cerebral Cortex, 13, 1369--1374.
• 9: Jasper, H.H. (1958). The ten-twenty electrode system of the international federation. Electroencephalography and Clinical Neurophysiology, 10, 371--375.
• 10: Jensen, O., Gelfand, J., Kounios, J., & Lisman, J.E. (2002). Oscillations in the alpha band (9-12 Hz) increase with memory load during retention in a short-term memory task. Cerebral Cortex, 12, 877--882
• 11: Kilmesch, W., Schimke, H., & Pfurtscheller, G. (1993). Alpha frequency, cognitive load and memory performance. Brain Topography, 5(3), 241--251.
• 12: Yoshifumi Kitamura , Yoshihisa Yamaguchi , Imamizu Hiroshi , Fumio Kishino , Mitsuo Kawato, Things happening in the brain while humans learn to use new tools, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642684]
• 13: J. Kohlmorgen , K.-R. Müller , K. Pawelzik, Analysis of drifting dynamics with neural network hidden Markov models, Proceedings of the 1997 conference on Advances in neural information processing systems 10, p.735-741, July 1998, Denver, Colorado, United States
• 14: Kramer, A.F. (1991). Physiological metrics of mental workload: A review of recent progress. In Multiple Task Performance (ed. Damos, D.L.), 279--328.
• 15: Johnny Chung Lee , Desney S. Tan, Using a low-cost electroencephalograph for task classification in HCI research, Proceedings of the 19th annual ACM symposium on User interface software and technology, October 15-18, 2006, Montreux, Switzerland [doi>10.1145/1166253.1166268]
• 16: David J. C. MacKay, Information Theory, Inference & Learning Algorithms, Cambridge University Press, New York, NY, 2002
• 17: Mecklinger, A., Kramer, A.F., & Strayer, D.L. (1992). Event related potentials and EEG components in a semantic memory search task. Psychophysiology, 29, 104--119.
• 18: Onton, J., Delorme, A., & Makeig, S. (2005). Frontal midline dynamics during working memory. Neuroimage, 27, 341--356.
• 19: Owen, A.M., McMillan, K.M., Laird, A.R., & Bullmore, E. (2005). N-back working memory paradigm: A meta-analysis of normative functional neuroimaging studies. Human Brain Mapping, 25, 46--59.
• 20: Palaniappan, R. (2005). Brain computer interface design using band powers extracted during mental tasks. Proceedings of the 2nd International IEEE EMBS Conference on Neural Engineering, 321--324.
• 21: Rosalind W. Picard, Affective computing, MIT Press, Cambridge, MA, 1997
• 22: Picton, T.W., Bentin, P., Berg, P., Hillyard, S.A., Johnson, J.R., Miller, G.A., et al. (2000). Guidelines for using human event-related potentials to study cognition: Recording standards and publication criteria. Psychophysiology, 37, 127--152.
• 23: Piolat, A., Olive, T., Roussey, J., Thunin, O., & Ziegler, J.C. (1999). SCRIPTKELL: A tool for measuring cognitive effort and time processing in writing and other complex cognitive activities. Behavior Research Methods, Instruments, & Computers, 31(1), 113--121.
• 24: Christopher W. Pleydell-Pearce , Sharron E. Whitecross , Blair T. Dickson, Multivariate Analysis of EEG: Predicting Cognition on the basis of Frequency Decomposition, Inter-electrode Correlation, Coherence, Cross Phase and Cross Power, Proceedings of the 36th Annual Hawaii International Conference on System Sciences (HICSS'03) - Track 5, p.131.1, January 06-09, 2003
• 25: III Lawrence J. Prinzel , Pope Alan T. , Freeman Frederick G. , Scerbo Mark W. , Mikulka Peter J., Empirical Analysis of EEG and ERPs for Pyschophysiological Adaptive Task Allocation, NASA Langley Technical Report Server, 2001
• 26: Rossion, B., & Pourtois, G. (2004). Revisiting Snodgrass and Vanderwart's object set: The role of surface detail in basic-level object recognition. Perception, 33, 217--236.
• 27: Smith, M.E., Gevins, A., Brown, H., Karnik, A., & Du, R. (2001). Monitoring task loading with multivariate EEG measures during complex forms of human-computer interaction. Human Factors, 43(3), 366--380.
• 28: Boris M. Velichkovsky , John Paulin Hansen, New technological windows into mind: there is more in eyes and brains for human-computer interaction, Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, p.496-503, April 13-18, 1996, Vancouver, British Columbia, Canada [doi>10.1145/238386.238619]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems

Additional Classification:
  B. Hardware
  B.4 INPUT/OUTPUT AND DATA COMMUNICATIONS
      B.4.2 Input/Output Devices
          Subjects: Channels and controllers

  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)

  J. Computer Applications
  J.3 LIFE AND MEDICAL SCIENCES

General Terms:
Human Factors

Keywords:
brain-computer interface, classification, cognitive load, electroencephalography (eeg), feature selection, machine learning, memory load