Semantic keyword-based retrieval of photos taken with mobile devices

ABSTRACT

This paper presents an approach for incorporating contextual metadata in a keyword-based photo retrieval process. We use our mobile annotation system PhotoMap in order to create metadata describing the photo shoot context (e.g., street address, nearby objects, season, lighting, nearby people...). These metadata are then used to generate a set of stamped words for indexing each photo. We adapt the Vector Space Model (VSM) in order to transform these shoot context words into document-vector terms. Furthermore, spatial reasoning is used for inferring new potential indexing terms. We define methods for weighting those terms and for handling a query matching. We also detail retrieval experiments carried out by using PhotoMap and Flickr geotagged photos. We illustrate the advantages of using Wikipedia georeferenced objects for indexing photos.

REFERENCES

• 1: Morgan Ames , Mor Naaman, Why we tag: motivations for annotation in mobile and online media, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240772]
• 2: Fergal Monaghan , David O'Sullivan, Automating Photo Annotation using Services and Ontologies, Proceedings of the 7th International Conference on Mobile Data Management, p.79, May 10-12, 2006 [doi>10.1109/MDM.2006.39]
• 3: Mor Naaman , Susumu Harada , QianYing Wang , Hector Garcia-Molina , Andreas Paepcke, Context data in geo-referenced digital photo collections, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027573]
• 4: Viana, W., Bringel Filho, J., Gensel, J., Villanova-Oliver, M., Martin, H. 2007. "PhotoMap: automatic spatiotemporal annotation for mobile photos". In Proceedings of the 7th International Symposium on Web and Wireless GIS (W2GIS) Cardiff, UK. Lecture Notes in Computer Science, Vol. 4857. 2007. DOI: http://dx.doi.org/10.1007/978-3-540-76925-5_14
• 5: N. O'Hare, H. Lee, S. Cooray, C. Gurrin, G. Jones, J. Malobabic, N. O'Connor, A. F. Smeaton, and B. Uscilowski. Mediassist: Using content- based analysis and context to manage personal photo collections. In CIVR, pages 529--532, Tempe, Arizona, USA, 2006.
• 6: Neil O'Hare , Cathal Gurrin , Gareth J. F. Jones , Hyowon Lee , Noel E. O'Connor , Alan F. Smeaton, Using text search for personal photo collections with the MediAssist system, Proceedings of the 2007 ACM symposium on Applied computing, March 11-15, 2007, Seoul, Korea [doi>10.1145/1244002.1244195]
• 7: Gregory D. Abowd , Elizabeth D. Mynatt, Charting past, present, and future research in ubiquitous computing, ACM Transactions on Computer-Human Interaction (TOCHI), v.7 n.1, p.29-58, March 2000 [doi>10.1145/344949.344988]
• 8: Joyce Y. Chai , Chen Zhang , Rong Jin, An empirical investigation of user term feedback in text-based targeted image search, ACM Transactions on Information Systems (TOIS), v.25 n.1, p.3-es, February 2007 [doi>10.1145/1198296.1198299]
• 9: Pablo Castells , Miriam Fernandez , David Vallet, An Adaptation of the Vector-Space Model for Ontology-Based Information Retrieval, IEEE Transactions on Knowledge and Data Engineering, v.19 n.2, p.261-272, February 2007 [doi>10.1109/TKDE.2007.22]
• 10: Hammiche, S.; Lopez, B.; Benbernou, S.; Hacid, M.-S.; "Query Rewriting for Semantic Multimedia Data Retrieval". Advances of Computational Intelligence in Industrial Systems p.351--p.372, 2008
• 11: G. Salton , A. Wong , C. S. Yang, A vector space model for automatic indexing, Communications of the ACM, v.18 n.11, p.613-620, Nov. 1975 [doi>10.1145/361219.361220]
• 12: D. Buscaldi, P. Rosso, and P. Peris. Inferring geographical ontologies from multiple resources for geographical information retrieval. In C. Jones and R. Purves, editors, Proceedings of 3rd SIGIR Workshop on Geographical Information Retrieval, August 2006.
• 13: Christopher B. Jones , Harith Alani , Douglas Tudhope, Geographical Information Retrieval with Ontologies of Place, Proceedings of the International Conference on Spatial Information Theory: Foundations of Geographic Information Science, p.322-335, September 19-23, 2001
• 14: Peng-Yuan Liu, Tie-Jun Zhao, Xiao-Feng Yu. Applicationoriented comparison and evaluation of six semantic similarity measures based on WordNet. International Conference on Machine Learning and Cybernetics, 2006
• 15: Jim Hendler, Web 3.0: Chicken Farms on the Semantic Web, Computer, v.41 n.1, p.106-108, January 2008 [doi>10.1109/MC.2008.34]
• 16: Ying Liu , Dengsheng Zhang , Guojun Lu , Wei-Ying Ma, A survey of content-based image retrieval with high-level semantics, Pattern Recognition, v.40 n.1, p.262-282, January, 2007 [doi>10.1016/j.patcog.2006.04.045]
• 17: Shuo Wang , Feng Jing , Jibo He , Qixing Du , Lei Zhang, IGroup: presenting web image search results in semantic clusters, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240718]
• 18: Tuffield, M., Harris, S., Dupplaw, D. P., et al., 2006. Image annotation with Photocopain. In Proceedings of Semantic Web Annotation of Multimedia (SWAMM-06) Workshop at the World Wide Web Conference 06. WWW.
• 19: Windson Viana , Rossana M. C. Andrade, XMobile: A MB-UID environment for semi-automatic generation of adaptive applications for mobile devices, Journal of Systems and Software, v.81 n.3, p.382-394, March, 2008 [doi>10.1016/j.jss.2007.04.045]
• 20: Voorhees, E. M. (1999). "The TREC-8 question answering track report". In Proceedings of the 8th Text Retrieval Conference, Gaithersburg, Maryland, USA, pp. 77--82.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.4 Hypertext/Hypermedia

General Terms:
Algorithms, Design, Experimentation

Keywords:
context awareness, index expansion, semantic image retrieval