Semantic manifold learning for image retrieval

ABSTRACT

Learning the user's semantics for CBIR involves two different sources of information: the similarity relations entailed by the content-based features, and the relevance relations specified in the feedback. Given that, we propose an augmented relation embedding (ARE) to map the image space into a semantic manifold that faithfully grasps the user's preferences. Besides ARE, we also look into the issues of selecting a good feature set for improving the retrieval performance. With these two aspects of efforts we have established a system that yields far better results than those previously reported. Overall, our approach can be characterized by three key properties: 1) The framework uses one relational graph to describe the similarity relations, and the other two to encode the relevant/irrelevant relations indicated in the feedback. 2) With the relational graphs so defined, learning a semantic manifold can be transformed into solving a constrained optimization problem, and is reduced to the ARE algorithm accounting for both the representation and the classification points of views. 3) An image representation based on augmented features is introduced to couple with the ARE learning. The use of these features is significant in capturing the semantics concerning different scales of image regions. We conclude with experimental results and comparisons to demonstrate the effectiveness of our method.

REFERENCES

• 1: Peter N. Belhumeur , João P. Hespanha , David J. Kriegman, Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.19 n.7, p.711-720, July 1997 [doi>10.1109/34.598228]
• 2: M. Belkin and P. Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. In Neural Information Processing Systems, 2001.
• 3: Y. Bengio, J.-F. Paiement, P. Vincent, O. Delalleau, N. Roux, and M. Ouimet. Out-of-sample extensions for lle, isomap, mds, eigenmaps, and spectral clustering. In Neural Information Processing Systems, 2003.
• 4: Chad Carson , Megan Thomas , Serge Belongie , Joseph M. Hellerstein , Jitendra Malik, Blobworld: A System for Region-Based Image Indexing and Retrieval, Proceedings of the Third International Conference on Visual Information and Information Systems, p.509-516, June 02-04, 1999
• 5: Hwann-Tzong Chen , Huang-Wei Chang , Tyng-Luh Liu, Local Discriminant Embedding and Its Variants, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, p.846-853, June 20-26, 2005 [doi>10.1109/CVPR.2005.216]
• 6: Thomas H. Cormen , Clifford Stein , Ronald L. Rivest , Charles E. Leiserson, Introduction to Algorithms, McGraw-Hill Higher Education, 2001
• 7: T. Cox and M. Cox. Multidimentional Scaling. Chapman & Hall, London, 1994.
• 8: Kristen Grauman , Trevor Darrell, Efficient Image Matching with Distributions of Local Invariant Features, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, p.627-634, June 20-26, 2005 [doi>10.1109/CVPR.2005.138]
• 9: Xiaofei He, Incremental semi-supervised subspace learning for image retrieval, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027530]
• 10: Xiaofei He , Wei-Ying Ma , Hong-Jiang Zhang, Learning an image manifold for retrieval, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027532]
• 11: X. He and P. Niyogi. Locality preserving projections. In Neural Information Processing Systems, 2003.
• 12: Chu-Hong Hoi , Michael R. Lyu, A novel log-based relevance feedback technique in content-based image retrieval, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027533]
• 13: Yoshiharu Ishikawa , Ravishankar Subramanya , Christos Faloutsos, MindReader: Querying Databases Through Multiple Examples, Proceedings of the 24rd International Conference on Very Large Data Bases, p.218-227, August 24-27, 1998
• 14: Yan Ke , Rahul Sukthankar , Larry Huston, An efficient parts-based near-duplicate and sub-image retrieval system, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027729]
• 15: David G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, International Journal of Computer Vision, v.60 n.2, p.91-110, November 2004 [doi>10.1023/B:VISI.0000029664.99615.94]
• 16: Wei-Ying Ma , B. S. Manjunath, NeTra: a toolbox for navigating large image databases, Multimedia Systems, v.7 n.3, p.184-198, May 1999 [doi>10.1007/s005300050121]
• 17: K. Mikolajczyk and C. Schmid. A performance evaluation of local descriptors. In Int'l Conference on Computer Vision and Pattern Recognition, pages 275--263, 2003.
• 18: A. Opelt, M. Fussenegger, A. Pinz, and P. Auer. Weak hypotheses and boosting for generic object detection and recognition. In Euro. Conference on Computer Vision, pages 71--84, 2004.
• 19: Greg Pass , Ramin Zabih , Justin Miller, Comparing images using color coherence vectors, Proceedings of the fourth ACM international conference on Multimedia, p.65-73, November 18-22, 1996, Boston, Massachusetts, United States [doi>10.1145/244130.244148]
• 20: Kriengkrai Porkaew , Kaushik Chakrabarti, Query refinement for multimedia similarity retrieval in MARS, Proceedings of the seventh ACM international conference on Multimedia (Part 1), p.235-238, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319463.319613]
• 21: S. Roweis and L. Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 290:2323--2326, 2000.
• 22: Yossi Rubner , Carlo Tomasi , Leonidas J. Guibas, The Earth Mover's Distance as a Metric for Image Retrieval, International Journal of Computer Vision, v.40 n.2, p.99-121, Nov. 2000 [doi>10.1023/A:1026543900054]
• 23: Y. Rui and T. Huang. Optimizing learning in image retrieval. In Int'l Conference on Computer Vision and Pattern Recognition, pages 236--243, 2000.
• 24: Y. Rui, T. Huang, and S. Mehrotra. Content-based image retrieval with relevance feedback in mars. In Int'l Conference on Image Processing, pages 815--818, 1997.
• 25: Gerard Salton , Michael J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, Inc., New York, NY, 1986
• 26: Josef Sivic , Andrew Zisserman, Video Google: A Text Retrieval Approach to Object Matching in Videos, Proceedings of the Ninth IEEE International Conference on Computer Vision, p.1470, October 13-16, 2003
• 27: J. Tenenbaum, V. de Silva, and J. Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 290:2319--2323, 2000.
• 28: K. Tieu and P. Viola. Boosting image retrieval. In Int'l Conference on Computer Vision and Pattern Recognition, pages 1228--1235, 2000.
• 29: Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500159]
• 30: V. Vapnik. Statistical Learning Theory. Wiley, 1998.
• 31: James Z. Wang , Jia Li , Gio Wiederhold, SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture LIbraries, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.9, p.947-963, September 2001 [doi>10.1109/34.955109]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Relevance feedback

General Terms:
Algorithms, Experimentation, Management, Performance

Keywords:
dimensionality reduction, feature selection, image retrieval, manifold learning, relevance feedback