The MIR flickr retrieval evaluation

ABSTRACT

In most well known image retrieval test sets, the imagery typically cannot be freely distributed or is not representative of a large community of users. In this paper we present a collection for the MIR community comprising 25000 images from the Flickr website which are redistributable for research purposes and represent a real community of users both in the image content and image tags. We have extracted the tags and EXIF image metadata, and also make all of these publicly available. In addition we discuss several challenges for benchmarking retrieval and classification methods.

REFERENCES

• 1: P. CLOUGH, H. MUELLER, AND M. SANDERSON. 2004. The CLEF cross-language image retrieval track (imageCLEF) 2004. In Fifth Workshop of the Cross-Language Evaluation Forum (CLEF 2004), LNCS, New York, NY, USA, ACM Press.
• 2: Michael S. Lew , Nicu Sebe , Chabane Djeraba , Ramesh Jain, Content-based multimedia information retrieval: State of the art and challenges, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.2 n.1, p.1-19, February 2006 [doi>10.1145/1126004.1126005]
• 3: S. MARCHAND-MAILLET. 2005. The Benchathlon Network. http://www.benchathlon.net.
• 4: Stéphane Marchand-Maillet , Marcel Worring, Benchmarking image and video retrieval: an overview, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178718]
• 5: Henning Müller , Stéphane Marchand-Maillet , Thierry Pun, The Truth about Corel - Evaluation in Image Retrieval, Proceedings of the International Conference on Image and Video Retrieval, p.38-49, July 18-19, 2002
• 6: X. ZHOU AND T. HUANG. 2003. Relevance feedback in image retrieval: a comprehensive review. ACM Multimedia Systems Journal, 8(6):536--544, April.
• 7: Alan F. Smeaton , Paul Over , Wessel Kraaij, Evaluation campaigns and TRECVid, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178722]
• 8: Clement H. C. Leung , Horace Ho-Shing Ip, Benchmarking for Content-Based Visual Information Search, Proceedings of the 4th International Conference on Advances in Visual Information Systems, p.442-456, November 02-04, 2000
• 9: Christian Fluhr , Pierre-Alain Moéllic , Patrick Hede, Usage-oriented multimedia information retrieval technological evaluation, Proceedings of the 8th ACM international workshop on Multimedia information retrieval, October 26-27, 2006, Santa Barbara, California, USA [doi>10.1145/1178677.1178719]
• 10: Mark J. Huiskes , Michael S. Lew, Performance evaluation of relevance feedback methods, Proceedings of the 2008 international conference on Content-based image and video retrieval, July 07-09, 2008, Niagara Falls, Canada [doi>10.1145/1386352.1386387]
• 11: K. SPARCK JONES AND C. VAN RIJSBERGEN (1975). Report on the need for and provision of an ideal information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge.
• 12: Pinaki Sinha , Ramesh Jain, Classification and annotation of digital photos using optical context data, Proceedings of the 2008 international conference on Content-based image and video retrieval, July 07-09, 2008, Niagara Falls, Canada [doi>10.1145/1386352.1386394]
• 13: J. YEN, P. WU, AND D. TRETTER (2007) Knowledge discovery for better photographs, Proc. SPIE 6506, 65060B.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.7 Digital Libraries
          Subjects: Collection

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Query formulation
      H.3.7 Digital Libraries
          Subjects: Dissemination; Standards

General Terms:
Experimentation, Human Factors, Measurement, Performance, Standardization

Keywords:
benchmarking, content-based image retrieval, image collections, relevance feedback