An exploratory study on joint analysis of visual classification in narrow domains and the discriminative power of tags

ABSTRACT

The popularity of social media sharing sites such as Flickr has driven a significant amount of research on the analysis of information contained in the tags used to annotate images. Many of such tags are not useful to describe the contents of an image and are often labeled as not descriptive or even noisy. In this work we focus on the descriptiveness of a tag in an exploratory way, within a relatively narrow domain, and with the help of a visual classifier. Preliminary experimental results demonstrate the possibility to infer descriptiveness of tags from a joint analysis of tag entropy calculations and the results of an automated visual classifier with a limited number of classes without taking tag content or tag co-occurrence into account. We postulate that these experiments can be extended and improved toward a working solution that might answer the question: Given a semantic category, which tags would you use for searching an image from that category?

REFERENCES

• 1: Morgan Ames , Mor Naaman, Why we tag: motivations for annotation in mobile and online media, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240772]
• 2: M. Aurnhammer, P. Hanappe, and L. Steels. Integrating collaborative tagging and emerging semantics for image retrieval. In Proceedings of the Collaborative Web Tagging Workshop (WWW 2006), 2006.
• 3: Shai Avidan , Ariel Shamir, Seam carving for content-aware image resizing, ACM SIGGRAPH 2007 papers, August 05-09, 2007, San Diego, California
• 4: M. R. Boutell , C. B. Brown , J. Luo, Review of the State of the Art in Semantic Scene Classification, University of Rochester, Rochester, NY, 2002
• 5: Ritendra Datta , Weina Ge , Jia Li , James Z. Wang, Toward Bridging the Annotation-Retrieval Gap in Image Search, IEEE MultiMedia, v.14 n.3, p.24-35, July 2007 [doi>10.1109/MMUL.2007.67]
• 6: Ritendra Datta , Dhiraj Joshi , Jia Li , James Z. Wang, Image retrieval: Ideas, influences, and trends of the new age, ACM Computing Surveys (CSUR), v.40 n.2, p.1-60, April 2008 [doi>10.1145/1348246.1348248]
• 7: A. Friedman. Framing pictures: the role of knowledge in automatized encoding and memory for gist. J Exp Psychol Gen, 108(3):316--55, 1979.
• 8: George W. Furnas , Caterina Fake , Luis von Ahn , Joshua Schachter , Scott Golder , Kevin Fox , Marc Davis , Cameron Marlow , Mor Naaman, Why do tagging systems work?, CHI '06 extended abstracts on Human factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1125451.1125462]
• 9: Scott A. Golder , Bernardo A. Huberman, Usage patterns of collaborative tagging systems, Journal of Information Science, v.32 n.2, p.198-208, April 2006 [doi>10.1177/0165551506062337]
• 10: M. Guy and E. Tonkin. Folksonomies - tidying up tags? D-Lib Magazine, 12(1), January 2006. ISSN 1082--9873.
• 11: Paul Heymann , Georgia Koutrika , Hector Garcia-Molina, Can social bookmarking improve web search?, Proceedings of the international conference on Web search and web data mining, February 11-12, 2008, Palo Alto, California, USA [doi>10.1145/1341531.1341558]
• 12: E. Hyvonen, A. Styrman, and S. Saarela. Ontology-based image retrieval. In Towards the semantic web and web services, Proceedings of XML Finland 2002 Conference, pages 15--27, 2002.
• 13: Roman Kern , Michael Granitzer , Viktoria Pammer, Extending Folksonomies for Image Tagging, Proceedings of the 2008 Ninth International Workshop on Image Analysis for Multimedia Interactive Services, p.126-129, May 07-09, 2008 [doi>10.1109/WIAMIS.2008.43]
• 14: J. Luo and M. Boutell. Natural scene classification using overcomplete ICA. Pattern Recognition, 38(10):1507--1519, 2005.
• 15: P. Mika. Ontologies are us: A unified model of social networks and semantics. In International Semantic Web Conference, LNCS, pages 522--536. Springer, 2005.
• 16: Oded Nov , Mor Naaman , Chen Ye, What drives content tagging: the case of photos on Flickr, Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, April 05-10, 2008, Florence, Italy [doi>10.1145/1357054.1357225]
• 17: Aude Oliva , Antonio Torralba, Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope, International Journal of Computer Vision, v.42 n.3, p.145-175, May-June 2001 [doi>10.1023/A:1011139631724]
• 18: A. Payne and S. Singh. Indoor vs. outdoor scene classification in digital photographs. Pattern Recognition, 38(10):1533--1545, 2005.
• 19: N. Rasiwasia, N. Vasconcelos, and P. Moreno. Query by semantic example. Proceedings of the International Conference in Image and Video Retrieval, pages 51--60, 2006.
• 20: Tye Rattenbury , Nathaniel Good , Mor Naaman, Towards automatic extraction of event and place semantics from flickr tags, Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, July 23-27, 2007, Amsterdam, The Netherlands [doi>10.1145/1277741.1277762]
• 21: Börkur Sigurbjörnsson , Roelof van Zwol, Flickr tag recommendation based on collective knowledge, Proceeding of the 17th international conference on World Wide Web, April 21-25, 2008, Beijing, China [doi>10.1145/1367497.1367542]
• 22: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972]
• 23: R. Troncy, J. van Ossenbruggen, J. Z. Pan, and G. Stamou. Image annotation on the semantic web. Technical report, World Wide Web Consortium, August 2007.
• 24: Roelof van Zwol, Flickr: Who is Looking?, Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence, p.184-190, November 02-05, 2007 [doi>10.1109/WI.2007.60]
• 25: Luis von Ahn, Games with a Purpose, Computer, v.39 n.6, p.92-94, June 2006 [doi>10.1109/MC.2006.196]
• 26: Roger C. F. Wong , Clement H. C. Leung, Automatic Semantic Annotation of Real-World Web Images, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.30 n.11, p.1933-1944, November 2008 [doi>10.1109/TPAMI.2008.125]
• 27: Xian Wu , Lei Zhang , Yong Yu, Exploring social annotations for the semantic web, Proceedings of the 15th international conference on World Wide Web, May 23-26, 2006, Edinburgh, Scotland [doi>10.1145/1135777.1135839]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Experimentation

Keywords:
image annotation, image retrieval, scene classification, social software, tag analysis