Take a look at the new version of this page: [ beta version ]. Tell us what you think.

ABSTRACT

Human visual perception is able to recognize a wide range of targets under challenging conditions, but has limited throughput. Machine vision and automatic content analytics can process images at a high speed, but suffers from inadequate recognition accuracy for general target classes. In this paper, we propose a new paradigm to explore and combine the strengths of both systems. A single trial EEG-based brain machine interface (BCI) subsystem is used to detect objects of interest of arbitrary classes from an initial subset of images. The EEG detection outcomes are used as input to a graph-based pattern mining subsystem to identify, refine, and propagate the labels to retrieve relevant images from a much larger pool. The combined strategy is unique in its generality, robustness, and high throughput. It has great potential for advancing the state of the art in media retrieval applications. We have evaluated and demonstrated significant performance gains of the proposed system with multiple and diverse image classes over several data sets, including those from Internet (Caltech 101) and remote sensing images. In this paper, we will also present insights learned from the experiments and discuss future research directions.

REFERENCES

• 1: N. Bigdely-Shamlo, A. Vankov, R. Ramirez, and S. Makeig. Brain Activity-Based Image Classification From Rapid Serial Visual Presentation. IEEE Trans. on NSRE, 16(5):432--441, 2008.
• 2: N. Bigdely-Shamlo, A. Vankov, R. R. Ramirez, and S. Makeig. Brain activity-based image classification from rapid serial visual presentation. IEEE Trans. on NSRE, 16(5):432--441, Oct. 2008.
• 3: J. Donoghue. Bridging the brain to the world: a perspective on neural interface systems. Neuron, 60(3):511--521, 2008.
• 4: M. Dyrholm, C. Christoforou, and L. Parra. Bilinear discriminant component analysis. JMLR, 8:1097--1111, 2007.
• 5: M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2008 (VOC2008) Results. http://www.pascalnetwork.org/challenges/VOC/voc2008/workshop/index.html.
• 6: Li Fei-Fei , Rob Fergus , Pietro Perona, Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories, Computer Vision and Image Understanding, v.106 n.1, p.59-70, April, 2007 [doi>10.1016/j.cviu.2005.09.012]
• 7: R. Fergus and P. Perona. A Visual Category Filter for Google Images. In Proc. ECCV, 2004.
• 8: A. Gerson, L. Parra, and P. Sajda. Cortically coupled computer vision for rapid image search. IEEE Trans. on NSRE, 14(2):174--179, 2006.
• 9: M. Gladwell. Blink: The power of thinking without thinking. Little, brown and company: Time warner book group, New York, 2005.
• 10: M. Hein and M. Maier. Manifold denoising. Proc. NIPS, 19, 2006.
• 11: T. Huang, C. Dagli, S. Rajaram, E. Chang, M. Mandel, G. Poliner, and D. Ellis. Active Learning for Interactive Multimedia Retrieval. Proc. of the IEEE, 96(4):648--667, 2008.
• 12: Tony Jebara , Jun Wang , Shih-Fu Chang, Graph construction and b-matching for semi-supervised learning, Proceedings of the 26th Annual International Conference on Machine Learning, p.441-448, June 14-18, 2009, Montreal, Quebec, Canada [doi>10.1145/1553374.1553432]
• 13: Yu-Gang Jiang , Chong-Wah Ngo , Jun Yang, Towards optimal bag-of-features for object categorization and semantic video retrieval, Proceedings of the 6th ACM international conference on Image and video retrieval, p.494-501, July 09-11, 2007, Amsterdam, The Netherlands [doi>10.1145/1282280.1282352]
• 14: Yushi Jing , Shumeet Baluja, VisualRank: Applying PageRank to Large-Scale Image Search, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.30 n.11, p.1877-1890, November 2008 [doi>10.1109/TPAMI.2008.121]
• 15: A. Kapoor, P. Shenoy, and D. Tan. Combining brain computer interfaces with vision for object categorization. In Proc. CVPR, 2008.
• 16: K. Kay, T. Naselaris, R. Prenger, and J. Gallant. Identifying natural images from human brain activity. Nature, 452(7185):352--355, 2008.
• 17: C. Keysers , D.-K. Xiao , P. Földiák , D. I. Perrett, The Speed of Sight, Journal of Cognitive Neuroscience, v.13 n.1, p.90-101, January 2001 [doi>10.1162/089892901564199]
• 18: John Langford , Lihong Li , Tong Zhang, Sparse Online Learning via Truncated Gradient, The Journal of Machine Learning Research, 10, p.777-801, June 2009
• 19: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 20: Charles A. Micchelli , Massimiliano Pontil, Learning the Kernel Function via Regularization, The Journal of Machine Learning Research, 6, p.1099-1125, 12/1/2005
• 21: Krystian Mikolajczyk , Cordelia Schmid, Scale & Affine Invariant Interest Point Detectors, International Journal of Computer Vision, v.60 n.1, p.63-86, October 2004 [doi>10.1023/B:VISI.0000027790.02288.f2]
• 22: Tom M. Mitchell , Rebecca Hutchinson , Radu S. Niculescu , Francisco Pereira , Xuerui Wang , Marcel Just , Sharlene Newman, Learning to Decode Cognitive States from Brain Images, Machine Learning, v.57 n.1-2, p.145-175, October-November 2004 [doi>10.1023/B:MACH.0000035475.85309.1b]
• 23: Y. Miyawaki, H. Uchida, O. Yamashita, M. Sato, Y. Morito, H. Tanabe, N. Sadato, and Y. Kamitani. Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders. Neuron, 60(5):915--929, 2008.
• 24: A. Oliva. Gist of the scene. In Encyclopedia of Neurobiology of Attention, pages 251--256, San Diego, CA, 2005. Elsevier.
• 25: L. Parra, C. Christoforou, A. Gerson, M. Dyrholm, A. Luo, M. Wagner, M. Philiastides, and P. Sajda. Spatiotemporal linear decoding of brain state: Application to performance augmentation in high-throughput tasks. IEEE Signal Processing Magazine, 25(1):95--115, January 2008.
• 26: L. Parra, C. Christoforou, A. Gerson, M. Dyrholm, A. Luo, M. Wagner, M. Philiastides, and P. Sajda. Spatiotemporal linear decoding of brain state: Application to performance augmentation in high-throughput tasks. IEEE Signal Processing Magazine, 25(1):95--115, January 2008.
• 27: P. Poolman, R. Frank, P. Luu, S. Pederson, and D. Tucker. A single-trial analytic framework for EEG analysis and its application to target detection and classification. NeuroImage, 42(2):787--798, 2008.
• 28: M. Potter and E. Levy. Recognition memory for a rapid sequence of pictures. Journal of Experimental Psychology, 81(1):10, 1969.
• 29: Y. Rui, T. Huang, M. Ortega, and S. Mehrotra. Relevance feedback: a power tool for interactive content-based image retrieval. IEEE Trans. on CSVT, 8(5):644--655, 1998.
• 30: M. Sanderson and P. Clough. cross-language image retrieval track. http://imageclef.org/.
• 31: Pradeep Shenoy , Desney S. Tan, Human-aided computing: utilizing implicit human processing to classify images, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357188]
• 32: Alan F. Smeaton , Paul Over , Wessel Kraaij, Evaluation campaigns and TRECVid, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178722]
• 33: S. Thorpe, D. Fize, and C. Marlot. Speed of processing in the human visual system. Nature, 381(6582):520--522, 1996.
• 34: J. Wang, S. F. Chang, X. Zhou, and S. T. C. Wong. Active microscopic cellular image annotation by superposable graph transduction with imbalanced labels. In Proc. CVPR, 2008.
• 35: Jun Wang , Tony Jebara , Shih-Fu Chang, Graph transduction via alternating minimization, Proceedings of the 25th international conference on Machine learning, p.1144-1151, July 05-09, 2008, Helsinki, Finland [doi>10.1145/1390156.1390300]
• 36: J. Wang, Y.-G. Jiang, and S.-F. Chang. Label diagnosis through self tuning for web image search. In Proc. CVPR, 2009.
• 37: D. Zhou, J. Weston, A. Gretton, O. Bousquet, and B. Scholkopf. Ranking on data manifolds. In Proc. NIPS, 2004.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human information processing

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Algorithms, Design, Experimentation

Keywords:
brain computer interface, image annotation and search, noisy label refinement, visual pattern mining