Object fingerprints for content analysis with applications to street landmark localization

ABSTRACT

An object can be a basic unit for multimedia content analysis. Besides similarity among common objects, each object has its own unique characteristics which we cannot find in other surrounding objects in multimedia data. We call such unique characteristics object fingerprints. In this paper, we propose a novel approach to extract and match object fingerprints for multimedia content analysis. In particular, we focus on the problem of street landmark localization from images. Instead of modeling and matching a street landmark as a whole, our proposed approach extracts the landmark's object fingerprints in a given image and match to a new image or video in order to localize the landmark. We formulate matching the landmark's object fingerprints as a classification problem solved by a cascade of 1NN classifiers. We develop a street landmark localization system that combines salient region detection, segmentation, and object fingerprint extraction techniques for the purpose. To evaluate, we have compiled a novel dataset which consists of 15 U.S. street landmarks' images and videos. Our experiments on this dataset show superior performance to state-of-the-art recognition algorithms [20, 33]. The proposed approach can also be well generalized to other objects of interest and content analysis tasks. We demonstrate the feasibility through the application of our approach to refine web image search results and obtained encouraging results.

REFERENCES

• 1: S. Belongie , J. Malik , J. Puzicha, Shape Matching and Object Recognition Using Shape Contexts, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.4, p.509-522, April 2002 [doi>10.1109/34.993558]
• 2: G. E. Burnett. Turn right at the king's head: Drivers' requirements for route guidance information. In PhD thesis, Loughborough University, UK, 1998.
• 3: Dorin Comaniciu , Peter Meer, Mean Shift: A Robust Approach Toward Feature Space Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.5, p.603-619, May 2002 [doi>10.1109/34.1000236]
• 4: Ritendra Datta , Dhiraj Joshi , Jia Li , James Z. Wang, Image retrieval: Ideas, influences, and trends of the new age, ACM Computing Surveys (CSUR), v.40 n.2, p.1-60, April 2008 [doi>10.1145/1348246.1348248]
• 5: A. de la Escalera, L. E. Moreno, M. A. Salichs, and J. M. Armingol. Road traffic sign detection and classification. IEEE Transactions on Industrial Electronics, 1997.
• 6: Pedro F. Felzenszwalb , Daniel P. Huttenlocher, Efficient Graph-Based Image Segmentation, International Journal of Computer Vision, v.59 n.2, p.167-181, September 2004 [doi>10.1023/B:VISI.0000022288.19776.77]
• 7: Pedro F. Felzenszwalb , Daniel P. Huttenlocher, Pictorial Structures for Object Recognition, International Journal of Computer Vision, v.61 n.1, p.55-79, January 2005 [doi>10.1023/B:VISI.0000042934.15159.49]
• 8: Andras Ferencz , Erik G. Learned-Miller , Jitendra Malik, Learning to Locate Informative Features for Visual Identification, International Journal of Computer Vision, v.77 n.1-3, p.3-24, May 2008 [doi>10.1007/s11263-007-0093-5]
• 9: Derek Hoiem , Alexei A. Efros , Martial Hebert, Putting Objects in Perspective, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.2137-2144, June 17-22, 2006 [doi>10.1109/CVPR.2006.232]
• 10: X. Hou and L. Zhang. Saliency detection: a spectral residual approach. In CVPR, 2007.
• 11: Winston H. Hsu , Lyndon S. Kennedy , Shih-Fu Chang, Video search reranking via information bottleneck principle, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180654]
• 12: Feng Jing , Changhu Wang , Yuhuan Yao , Kefeng Deng , Lei Zhang , Wei-Ying Ma, IGroup: web image search results clustering, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180720]
• 13: Yan Ke , Rahul Sukthankar , Larry Huston, An efficient parts-based near-duplicate and sub-image retrieval system, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027729]
• 14: Lyndon Kennedy , Mor Naaman , Shane Ahern , Rahul Nair , Tye Rattenbury, How flickr helps us make sense of the world: context and content in community-contributed media collections, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291384]
• 15: S. Kim, X. Jin, and J. Han. Sparclus: Spatial relationship pattern-based hierarchical clustering. In SIAM Int. Conf. on Data Mining, 2008.
• 16: Bart Kroon , Sabri Boughorbel , Alan Hanjalic, Person-based search in videos, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291263]
• 17: B. Leibe, N. Cornelis, K. Cornelis, and L. Van Gool. Dynamic 3d scene analysis from a moving vehicle. In CVPR, 2007.
• 18: Michael S. Lew , Nicu Sebe , Chabane Djeraba , Ramesh Jain, Content-based multimedia information retrieval: State of the art and challenges, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.2 n.1, p.1-19, February 2006 [doi>10.1145/1126004.1126005]
• 19: Yiqun Li , Joo Hwee Lim, Outdoor place recognition using compact local descriptors and multiple queries with user verification, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291368]
• 20: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 21: T. Malisiewicz and A. A. Efros. Improving spatial support for objects via multiple segmentations. In British Machine Vision Conference, 2007.
• 22: Humera Noor , Shahid H. Mirza , Yaser Sheikh , Amit Jain , Mubarak Shah, Model generation for video-based object recognition, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180791]
• 23: G. Piccioli, E. De Micheli, P. Parodi, and M. Campani. Robust method for road sign detection and recognition. Image and Vision Computing, 1996.
• 24: J. Pilet, V. Lepetit, and P. Fua. Real-time non-rigid surface detection. In CVPR, 2005.
• 25: Arun Qamra , Edward Y. Chang, Scalable landmark recognition using EXTENT, Multimedia Tools and Applications, v.38 n.2, p.187-208, June 2008 [doi>10.1007/s11042-007-0178-8]
• 26: Subhajit Sanyal , Srinivasan H. Sengamedu, LogoSeeker: a system for detecting and matching logos in natural images, Proceedings of the 15th international conference on Multimedia, September 25-29, 2007, Augsburg, Germany [doi>10.1145/1291233.1291272]
• 27: G. Schindler, M. Brown, and R. Szeliski. City-scale location recognition. In CVPR, 2007.
• 28: Jianbo Shi , Jitendra Malik, Normalized Cuts and Image Segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.8, p.888-905, August 2000 [doi>10.1109/34.868688]
• 29: Josef Sivic , Andrew Zisserman, Video Google: A Text Retrieval Approach to Object Matching in Videos, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.1470, October 13-16, 2003
• 30: Antonio Torralba , Kevin P. Murphy , William T. Freeman , Mark A. Rubin, Context-based vision system for place and object recognition, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.273, October 13-16, 2003
• 31: J. Willamowski, D. Arregui, G. Csurka, C. R. Dance, and L. Fan. Categorizing nine visual classes using local appearance descriptors. In IWLAVS, 2004.
• 32: Wen Wu , Xilin Chen , Jie Yang, Incremental detection of text on road signs from video with application to a driving assistant system, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027724]
• 33: Wei Zhang , Jana Kosecka, Image Based Localization in Urban Environments, Proceedings of the Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06), p.33-40, June 14-16, 2006 [doi>10.1109/3DPVT.2006.80]
• 34: W. Zhao , R. Chellappa , P. J. Phillips , A. Rosenfeld, Face recognition: A literature survey, ACM Computing Surveys (CSUR), v.35 n.4, p.399-458, December 2003 [doi>10.1145/954339.954342]
• 35: Qing-Fang Zheng , Wei-Qiang Wang , Wen Gao, Effective and efficient object-based image retrieval using visual phrases, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180664]
• 36: C. L. Zitnick, J. Sun, R. Szeliski, and S. Winder. Object instance recognition using triplets of feature symbols. In Microsoft Research Technical Report, 2007.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Object recognition

General Terms:
Algorithms

Keywords:
object fingerprint, street landmark localization