Constructing visual phrases for effective and efficient object-based image retrieval

ABSTRACT

The explosion of multimedia data necessitates effective and efficient ways for us to get access to our desired ones. In this article, we draw an analogy between image retrieval and text retrieval and propose a visual phrase-based approach to retrieve images containing desired objects (object-based image retrieval). The visual phrase is defined as a pair of frequently co-occurred adjacent local image patches and is constructed using data mining. We design methods on how to construct visual phrase and how to index/search images based on visual phrase. We demonstrate experiments to show our visual phrase-based approach can be very efficient and more effective than current visual word-based approach.

REFERENCES

• 1: Rakesh Agrawal , Ramakrishnan Srikant, Fast Algorithms for Mining Association Rules in Large Databases, Proceedings of the 20th International Conference on Very Large Data Bases, p.487-499, September 12-15, 1994
• 2: Shivani Agarwal , Aatif Awan , Dan Roth, Learning to Detect Objects in Images via a Sparse, Part-Based Representation, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.26 n.11, p.1475-1490, November 2004 [doi>10.1109/TPAMI.2004.108]
• 3: Chad Carson , Serge Belongie , Hayit Greenspan , Jitendra Malik, Blobworld: Image Segmentation Using Expectation-Maximization and Its Application to Image Querying, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.8, p.1026-1038, August 2002 [doi>10.1109/TPAMI.2002.1023800]
• 4: Aaron Ceglar , John F. Roddick, Association mining, ACM Computing Surveys (CSUR), v.38 n.2, p.5-es, 2006 [doi>10.1145/1132956.1132958]
• 5: Cox, I. J., Miller, M. L., Minka, T. P., Papathomas, T. V., and Yianilos, P. N. 2000. The Bayesian image retrieval system, PicHunter: theory, implementation, and psychophysical experiments. IEEE Trans. Image Process. 9, 1, 20--37.
• 6: Datta, R., Joshi, D., Li, J., and Wang, J. 2006. Image retrieval: ideas, influences, and trends of the new age. Penn Sate University Tech. rep. CSE 06--009.
• 7: Enser, P. G. B. and Sandom, C. J. 2003. Towards a comprehensive survey of the semantic gap in visual image retrieval. In Proceedings of the 2nd International Conference on Image and video retrieval. E. M. Bakker, T. S. Huang, M. S. Lew, N. Sebe, and X. Zhou, Eds. Springer-Verlag, London, UK. 291--299
• 8: R. Fergus , L. Fei-Fei , P. Perona , A. Zisserman, Learning Object Categories from Google"s Image Search, Proceedings of the Tenth IEEE International Conference on Computer Vision, p.1816-1823, October 17-20, 2005 [doi>10.1109/ICCV.2005.142]
• 9: Myron Flickner , Harpreet Sawhney , Wayne Niblack , Jonathan Ashley , Qian Huang , Byron Dom , Monika Gorkani , Jim Hafner , Denis Lee , Dragutin Petkovic , David Steele , Peter Yanker, Query by Image and Video Content: The QBIC System, Computer, v.28 n.9, p.23-32, September 1995 [doi>10.1109/2.410146]
• 10: David Garcya-Perez , Antonio Mosquera , Stefano Berretti , Alberto Del Bimbo, Object-Based Image Retrieval Using Active Nets, Proceedings of the 18th International Conference on Pattern Recognition, p.750-753, August 20-24, 2006 [doi>10.1109/ICPR.2006.887]
• 11: Goethals, B. 2003. Survey on frequent pattern mining. Helsinki Institute for Information Technology tech. rep.
• 12: Grabner, M., Grabner, H., and Bischof, H. 2006. Fast Approximated SIFT. In Proceedings of 7th Asian Conference on Computer Vision, P. J. Narayanan, S. K. Nayar, H.--Y. Shum, Eds. Springer-Verlag, London, UK. 918--927.
• 13: Khaled M. Hammouda , Mohamed S. Kamel, Efficient Phrase-Based Document Indexing for Web Document Clustering, IEEE Transactions on Knowledge and Data Engineering, v.16 n.10, p.1279-1296, October 2004 [doi>10.1109/TKDE.2004.58]
• 14: Hoiem, R., Sukthankar, H., Schneiderman, H., and Huston, L. 2004. Object-based image retrieval using the statistical structure of images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, Washington, DC. Vol. 2, 490--497.
• 15: Bernard J. Jansen , Amanda Spink , Tefko Saracevic, Real life, real users, and real needs: a study and analysis of user queries on the web, Information Processing and Management: an International Journal, v.36 n.2, p.207-227, Jan.1.2000 [doi>10.1016/S0306-4573(99)00056-4]
• 16: Rong Jin , Joyce Y. Chai , Luo Si, Effective automatic image annotation via a coherent language model and active learning, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027732]
• 17: Jing, F., Li, M., Zhang, H.-J., and Zhang, B. 2004, An efficient and effective region-based image retrieval framework. IEEE Trans. Image Process. 13, 5, 699--709.
• 18: Frederic Jurie , Bill Triggs, Creating Efficient Codebooks for Visual Recognition, Proceedings of the Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, p.604-610, October 17-20, 2005 [doi>10.1109/ICCV.2005.66]
• 19: Yan Ke , Rahul Sukthankar , Larry Huston, An efficient parts-based near-duplicate and sub-image retrieval system, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027729]
• 20: M. L. Kherfi , D. Ziou , A. Bernardi, Image Retrieval from the World Wide Web: Issues, Techniques, and Systems, ACM Computing Surveys (CSUR), v.36 n.1, p.35-67, March 2004 [doi>10.1145/1013208.1013210]
• 21: Michael S. Lew , Nicu Sebe , Chabane Djeraba , Ramesh Jain, Content-based multimedia information retrieval: State of the art and challenges, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.2 n.1, p.1-19, February 2006 [doi>10.1145/1126004.1126005]
• 22: Jia Li , James Z. Wang, Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.9, p.1075-1088, September 2003 [doi>10.1109/TPAMI.2003.1227984]
• 23: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 24: Jie Luo , Mario A. Nascimento, Content based sub-image retrieval via hierarchical tree matching, Proceedings of the 1st ACM international workshop on Multimedia databases, November 07-07, 2003, New Orleans, LA, USA [doi>10.1145/951676.951689]
• 25: Marcin Marszaek , Cordelia Schmid, Spatial Weighting for Bag-of-Features, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2118-2125, June 17-22, 2006 [doi>10.1109/CVPR.2006.288]
• 26: Krystian Mikolajczyk , Cordelia Schmid, A Performance Evaluation of Local Descriptors, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.10, p.1615-1630, October 2005 [doi>10.1109/TPAMI.2005.188]
• 27: Caroline Pantofaru , Gyuri Dorko , Cordelia Schmid , Martial Hebert, Combining Regions and Patches for Object Class Localization, Proceedings of the 2006 Conference on Computer Vision and Pattern Recognition Workshop, p.23, June 17-22, 2006 [doi>10.1109/CVPRW.2006.57]
• 28: Constantine P. Papageorgiou , Michael Oren , Tomaso Poggio, A General Framework for Object Detection, Proceedings of the Sixth International Conference on Computer Vision, p.555, January 04-07, 1998
• 29: A. Pentland , R. W. Picard , S. Sclaroff, Photobook: content-based manipulation of image databases, International Journal of Computer Vision, v.18 n.3, p.233-254, June 1996 [doi>10.1007/BF00123143]
• 30: Rui, Y., Huang, T. S., and Chang, S.--F. 1999. Image retrieval: Current techniques, promising directions and open Issues. J. Visual Comm. Image Represent. 10, 1, 39--62.
• 31: Nicu Sebe , Michael S. Lew , Dionysius P. Huijsmans, Multi-scale sub-image search, Proceedings of the seventh ACM international conference on Multimedia (Part 2), p.79-82, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319878.319901]
• 32: Josef Sivic , Andrew Zisserman, Video Google: A Text Retrieval Approach to Object Matching in Videos, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.1470, October 13-16, 2003
• 33: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972]
• 34: John R. Smith , Shih-fu Chang, Querying by color regions using VisualSEEk content-based visual query system, Intelligent multimedia information retrieval, MIT Press, Cambridge, MA, 1997
• 35: Smyth, W. F., Lam, C. P., Chen, X., and Maxville, V. 2003. Heuristics for image retrieval using spatial configurations. In Proceedings of the 7th Digital Image Computing: Techniques and Applications. Sydney. 909--918.
• 36: David McG. Squire , Wolfgang Müller , Henning Müller , Thierry Pun, Content-based query of image databases: inspirations from text retrieval, Pattern Recognition Letters, v.21 n.13-14, p.1193-1198, Dec. 2000 [doi>10.1016/S0167-8655(00)00081-7]
• 37: Zehang Sun , George Bebis , Ronald Miller, On-Road Vehicle Detection: A Review, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.28 n.5, p.694-711, May 2006 [doi>10.1109/TPAMI.2006.104]
• 38: Kah-Kay Sung , Tomaso Poggio, Example-Based Learning for View-Based Human Face Detection, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.1, p.39-51, January 1998 [doi>10.1109/34.655648]
• 39: Veltkamp, R. C. and Tanase, M. 2000. Content-based image retrieval systems: A survey. Utrecht University technical rep. UU-CS-2000-34.
• 40: Viola, P. and Jones, M. 2001. Rapid object detection using a boosted cascade of simple features. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, Washington, DC. 511--518.
• 41: Gang Wang , Ye Zhang , Li Fei-Fei, Using Dependent Regions for Object Categorization in a Generative Framework, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.1597-1604, June 17-22, 2006 [doi>10.1109/CVPR.2006.324]
• 42: Hugh E. Williams , Justin Zobel , Dirk Bahle, Fast phrase querying with combined indexes, ACM Transactions on Information Systems (TOIS), v.22 n.4, p.573-594, October 2004 [doi>10.1145/1028099.1028102]
• 43: J. Winn , A. Criminisi , T. Minka, Object Categorization by Learned Universal Visual Dictionary, Proceedings of the Tenth IEEE International Conference on Computer Vision, p.1800-1807, October 17-20, 2005 [doi>10.1109/ICCV.2005.171]
• 44: Ian H. Witten , Alistair Moffat , Timothy C. Bell, Managing gigabytes (2nd ed.): compressing and indexing documents and images, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1999
• 45: Xifeng Yan , Philip S. Yu , Jiawei Han, Graph indexing based on discriminative frequent structure analysis, ACM Transactions on Database Systems (TODS), v.30 n.4, p.960-993, December 2005 [doi>10.1145/1114244.1114248]
• 46: Ming-Hsuan Yang , David J. Kriegman , Narendra Ahuja, Detecting Faces in Images: A Survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.1, p.34-58, January 2002 [doi>10.1109/34.982883]
• 47: Yeh, T., Tollmar, K., and Darrell, T. 2004. Searching the Web with mobile images for location recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, Washington, DC. Vol. 2, 76--81.
• 48: Qing-Fang Zheng , Wei-Qiang Wang , Wen Gao, Effective and efficient object-based image retrieval using visual phrases, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180664]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Image databases

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

General Terms:
Algorithms, Experimentation, Performance

Keywords:
Content-based image retrieval, SIFT, inverted index, local image descriptor, object-based image retrieval, visual phrase