Adaptive image retrieval using a Graph model for semantic feature integration

ABSTRACT

The variety of features available to represent multimedia data constitutes a rich pool of information. However, the plethora of data poses a challenge in terms of feature selection and integration for effective retrieval. Moreover, to further improve effectiveness, the retrieval model should ideally incorporate context-dependent feature representations to allow for retrieval on a higher semantic level. In this paper we present a retrieval model and learning framework for the purpose of interactive information retrieval. We describe how semantic relations between multimedia objects based on user interaction can be learnt and then integrated with visual and textual features into a unified framework. The framework models both feature similarities and semantic relations in a single graph. Querying in this model is implemented using the theory of random walks. In addition, we present ideas to implement short-term learning from relevance feedback. Systematic experimental results validate the effectiveness of the proposed approach for image retrieval. However, the model is not restricted to the image domain and could easily be employed for retrieving multimedia data (and even a combination of different domains, eg images, audio and text documents).

REFERENCES

• 1: Sergey Brin , Lawrence Page, The anatomy of a large-scale hypertextual Web search engine, Computer Networks and ISDN Systems, v.30 n.1-7, p.107-117, April 1, 1998
• 2: Ronald Fagin , Ravi Kumar , D. Sivakumar, Efficient similarity search and classification via rank aggregation, Proceedings of the 2003 ACM SIGMOD international conference on Management of data, June 09-12, 2003, San Diego, California [doi>10.1145/872757.872795]
• 3: J. Han, M. Li, H. Zhang, and L. Guo. A memory learning framework for effective image retrieval. IEEE Trans. Image Processing, 14(4): 511--524, 2005.
• 4: Jingrui He , Hanghang Tong , Mingjing Li , Wei-Ying Ma , Changshui Zhang, Multiple random walk and its application in content-based image retrieval, Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval, November 10-11, 2005, Hilton, Singapore [doi>10.1145/1101826.1101852]
• 5: M.-K. Hu. Visual pattern recognition by moment invariants. IEEE Trans. Information Theory, 8(2): 179--187, Feb. 1962.
• 6: G. Iyengar , P. Duygulu , S. Feng , P. Ircing , S. P. Khudanpur , D. Klakow , M. R. Krause , R. Manmatha , H. J. Nock , D. Petkova , B. Pytlik , P. Virga, Joint visual-text modeling for automatic retrieval of multimedia documents, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101154]
• 7: J. Jeon , V. Lavrenko , R. Manmatha, Automatic image annotation and retrieval using cross-media relevance models, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860459]
• 8: A. N. Langville and C. D. Meyer. Deeper inside pagerank. Internet Mathematics, 1(3):335--400, 2004.
• 9: Dik L. Lee , Huei Chuang , Kent Seamons, Document Ranking and the Vector-Space Model, IEEE Software, v.14 n.2, p.67-75, March 1997 [doi>10.1109/52.582976]
• 10: L. Lovasz. Random walks on graphs: A survey. Combinatorics, Paul Erdos is Eighty, 2:353--398, 1993.
• 11: NIST. Proc. of the TREC Video Retrieval Evaluation Conference (TRECVID2005), Gaithersburgh, MD, USA, Nov. 2005.
• 12: Jia-Yu Pan , Hyung-Jeong Yang , Christos Faloutsos , Pinar Duygulu, GCap: Graph-based Automatic Image Captioning, Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04) Volume 9, p.146, June 27-July 02, 2004
• 13: J. J. Rocchio. Relevance feedback in information retrieval. In The SMART retrieval system: experiments in automatic document processing, pages 313--323. Prentice-Hall, Englewood Cliffs, US, 1971.
• 14: Y. Rui and T. S. Huang. Optimizing learning in image retrieval. In IEEE Proc. of Conf. on Computer Vision and Pattern Recognition (CVPR-00), pages 236--245, Los Alamitos, June 2000.
• 15: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972]
• 16: M. Sonka, V. Hlavac, and R. Boyle. Image Processing, Analysis, and Machine Vision. Thomson-Engineering, Toronto, Canada, 2nd edition, 1998.
• 17: M. Stricker and M. Orengo. Similarity of color images. In Proc. of the SPIE: Storage and Retrieval for Image and Video Databases, volume 2420, pages 381--392, Feb. 1995.
• 18: HongJiang Zhang , Zhong Su, Relevance Feedback in CBIR, Proceedings of the IFIP TC2/WG2.6 Sixth Working Conference on Visual Database Systems: Visual and Multimedia Information Management, p.21-35, May 29-31, 2002
• 19: Hanghang Tong , Jingrui He , Mingjing Li , Changshui Zhang , Wei-Ying Ma, Graph based multi-modality learning, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101337]
• 20: J. Urban and J. M. Jose. EGO: A personalised multimedia management tool. In Proc. of the 2nd Int. Workshop on Adaptive Multimedia Retrieval, pages 3--17, 2004.
• 21: Jana Urban , Joemon M. Jose, Evidence Combination for Multi-Point Query Learning in Content-Based Image Retrieval, Proceedings of the IEEE Sixth International Symposium on Multimedia Software Engineering (ISMSE'04), p.583-586, December 13-15, 2004
• 22: J. Urban and J. M. Jose. Evaluating a workspace's usefulness for image retrieval. ACM Multimedia Sytems Journal (Special Issue on User-Centered Multimedia), 2006. accepted for publication.
• 23: C. J. Van Rijsbergen, Information Retrieval, Butterworth-Heinemann, Newton, MA, 1979
• 24: Berkley's list of Corel CDs, and image annotations. Available from http://elib.cs.berkeley.edu/corel/, last accessed in July 2006.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Relevance feedback

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Experimentation, Performance

Keywords:
fusion, image retrieval, random walks, relevance feedback, retrieval models, semantic features