Human-aided computing: utilizing implicit human processing to classify images

ABSTRACT

In this paper, we present Human-Aided Computing, an approach that uses an electroencephalograph (EEG) device to measure the presence and outcomes of implicit cognitive processing, processing that users perform automatically and may not even be aware of. We describe a classification system and present results from two experiments as proof-of-concept. Results from the first experiment showed that our system could classify whether a user was looking at an image of a face or not, even when the user was not explicitly trying to make this determination. Results from the second experiment extended this to animals and inanimate object categories as well, suggesting generality beyond face recognition. We further show that we can improve classification accuracies if we show images multiple times, potentially to multiple people, attaining well above 90% classification accuracies with even just ten presentations.

REFERENCES

• 1: Amazon.com. Mechanical Turk. http://www.mturk.com.
• 2: David P. Anderson , Jeff Cobb , Eric Korpela , Matt Lebofsky , Dan Werthimer, SETI@home: an experiment in public-resource computing, Communications of the ACM, v.45 n.11, p.56-61, November 2002 [doi>10.1145/581571.581573]
• 3: Broadbent, D.E. (1958). Perception and communication. Pergamon Press: London, UK.
• 4: Carmel, D., Bentin, S. (2002). Domain specificity versus expertise: factors influencing distinct processing of faces. Journal of Cognition, 83, 1--29.
• 5: Farwell, L.A., Donchin, E. (1988). Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalography & Clinical Neurophysiology, 70(6), 510--23.
• 6: Fisch, B.J. (2005). Fisch & Spehlmann's EEG primer: Basic principles of digital and analog EEG. Elsevier: Amsterdam.
• 7: Gerson, A.D., Parra, L.C., & Sajda, P. (2006). Cortically-coupled computer vision for rapid image search. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 14(2), 174--179.
• 8: Griffin, G. Holub, A.D. Perona, P. (2007). The Caltech-256, Caltech Technical Report.
• 9: Grill-Spector, K. (2003). The neural basis of object perception. Current opinion in neurobiology, 13, 1--8.
• 10: Salim Hariri , Manish Parashar, Tools and Environments for Parallel and Distributed Computing, Wiley-Interscience, 2004
• 11: Hastie, T., Tibshirani, R. & Friedman, J.H. (2001). The elements of statistical learning: Data mining, inference, and prediction. Springer Series in Statistics. Springer-Verlag: New York, New York.
• 12: Hoffman, U., Vesin, J-M, Ebrahimi, T. (2006). Spatial filters for the classification of event-related potentials. In European Symposium on Artificial Neural Networks, 47--52.
• 13: Itier, R.J., Taylor, M.J. (2004). N170 or N1? Spatiotemporal differences between object and face processing using ERPs. Cerebral Cortex, 14, 132--142.
• 14: Jasper, H.H. (1958). The ten-twenty electrode system of the international federation. Electroencephalography and Clinical Neurophysiology, 10, 371--375.
• 15: Johnson J.S., Olshausen, B.A. (2005). The earliest EEG signatures of object recognition in a cued-target task are postsensory. Journal of Vision, 5(4), 299--312.
• 16: Koch, C., Tsuchiya, N. (2006). Attention and consciousness: Two distinct brain processes. Trends in Cognitive Sciences, 11(1), 16--22.
• 17: Larson, S.M., Snow, C., & Pande, V.S. (2003). Folding@Home and Genome@Home: Using distributed computing to tackle previously intractable problems in computational biology. Modern Methods in Computational Biology (ed. R. Grant). Horizon Press.
• 18: Johnny Chung Lee , Desney S. Tan, Using a low-cost electroencephalograph for task classification in HCI research, Proceedings of the 19th annual ACM symposium on User interface software and technology, October 15-18, 2006, Montreux, Switzerland [doi>10.1145/1166253.1166268]
• 19: J. Zhang , M. Marszałek , S. Lazebnik , C. Schmid, Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive Study, International Journal of Computer Vision, v.73 n.2, p.213-238, June 2007 [doi>10.1007/s11263-006-9794-4]
• 20: Miller, G.A. (1987) Psychology: the science of mental life. Penguin: Harmondsworth, UK.
• 21: Posner, K.R. (1978). Chronometric explorations of mind. Lawrence Erlbaum Associates: Hillsdale, NJ.
• 22: Rossion, B., Gauthier, I., Delvenne, J.-F., Tarr, M.J., Bruyer, R., & Crommelinck, M. (1999). Does the N170 occipito-temporal component reflect a face-specific structural encoding stage? Poster at Object Perception and Memory 1999.
• 23: Velmans, M. (1991). Is human processing conscious? Behavioral and Brain Sciences, 14, 651--726.
• 24: Luis von Ahn, Games with a Purpose, Computer, v.39 n.6, p.92-94, June 2006 [doi>10.1109/MC.2006.196]
• 25: Luis von Ahn , Laura Dabbish, Labeling images with a computer game, Proceedings of the SIGCHI conference on Human factors in computing systems, p.319-326, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985733]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems

Additional Classification:
  B. Hardware
  B.4 INPUT/OUTPUT AND DATA COMMUNICATIONS
      B.4.2 Input/Output Devices

  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Input devices and strategies (e.g., mouse, touchscreen)

  J. Computer Applications
  J.3 LIFE AND MEDICAL SCIENCES

General Terms:
Human Factors

Keywords:
brain-computer interface, electroencephalography (eeg), human cognition, image classification, implicit processing, visual attention