Annotation suggestion and search for personal multimedia objects on the web

ABSTRACT

The number of personal multimedia objects, such as digital photographs and videos, are exploding on the web through popular sites such as Flickr, YouTube, and FaceBook hosting billions of user-created items. Semantic annotation can be an extremely effective way to search, browse, and organize media objects, but can require extensive human involvement. In this work, we show how semantic metadata about social networks and family relationships can be used to improve semantic annotation suggestion. This includes up to 82% recall for people annotations as well as recall improvements of 20-26% in tag annotation recall when no annotation history is available. In addition, utilizing relationships among people while searching can provide at least 28% higher recall and 55% higher precision than keyword search while still being up to 12 times faster. Methods are evaluated on real personal photo collections containing up to 120k photos from Flickr as well as 41k annotated photos from our prototype system.

REFERENCES

• 1: J. Golbeck, A. Alford, R. Alford, J. Hendler, "Organization and Structure of Information using Semantic Web Technologies," in Handbook of Human Factors in Web Design, Robert W. Proctor and Kim-Phuong L. Vu (eds.). Lawrence Erlbaum Associates, NJ: 2004. Available at: http://www.mindswap.org/papers/Handbook.pdf.
• 2: H. Hoang, A. Tjoa. "The State of the Art of Ontology-based Query Systems: A Comparison of Existing Approaches," Proceedings of the IEEE International Conference on Computing & Informatics (2006).
• 3: E. Hyvönen, S. Saarela and K. Viljanen: "Ontogator: Combining View- and Ontology-Based Search with Semantic Browsing." Proceedings of XML Finland 2003.
• 4: Ka-Ping Yee , Kirsten Swearingen , Kevin Li , Marti Hearst, Faceted metadata for image search and browsing, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642681]
• 5: L. Hollink, G. Schreiber, J. Wielemaker, and B. Wielinga. "Semantic annotation of image collections." In Workshop on Knowledge Markup and Semantic Annotation, KCAP'03, 2003.
• 6: A. Th. (Guus) Schreiber , Barbara Dubbeldam , Jan Wielemaker , Bob Wielinga, Ontology-Based Photo Annotation, IEEE Intelligent Systems, v.16 n.3, p.66-74, May 2001 [doi>10.1109/5254.940028]
• 7: K. Latif, K. Mustofa, A. Min Tjoa. "An Approach for a Personal Information Management System for Photos of a Lifetime by Exploiting Semantics." DEXA '06, 2006, pp. 467--477.
• 8: K. Latif, A. Min Tjoa. "Combining Context Ontology and Landmarks for Personal Information Management." In Proceedings of ICOCI, 2006.
• 9: Jia-Yu Pan, Hyung-Jeong Yang, Christos Faloutsos, and Pinar Duygulu. GCap: Graph-based Automatic Image Captioning. MDDE 2004.
• 10: Jia-Yu Pan , Hyung-Jeong Yang , Christos Faloutsos , Pinar Duygulu, Automatic multimedia cross-modal correlation discovery, Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, August 22-25, 2004, Seattle, WA, USA [doi>10.1145/1014052.1014135]
• 11: Jia Li , James Z. Wang, Real-time computerized annotation of pictures, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180841]
• 12: Xiaonan Lu , Prasenjit Mitra , James Z. Wang , C. Lee Giles, Automatic categorization of figures in scientific documents, Proceedings of the 6th ACM/IEEE-CS joint conference on Digital libraries, June 11-15, 2006, Chapel Hill, NC, USA [doi>10.1145/1141753.1141778]
• 13: Ebroul Izquierdo , Andres Dorado, Climbing the Semantic Ladder: Towards Semantic Semi--Automatic Image Annotation using MPEG--7 Descriptor Schemas, Proceedings of the Seventh International Workshop on Computer Architecture for Machine Perception, p.257-262, July 04-06, 2005 [doi>10.1109/CAMP.2005.16]
• 14: Mor Naaman , Ron B. Yeh , Hector Garcia-Molina , Andreas Paepcke, Leveraging context to resolve identity in photo albums, Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2005, Denver, CO, USA [doi>10.1145/1065385.1065430]
• 15: Adrian Graham , Hector Garcia-Molina , Andreas Paepcke , Terry Winograd, Time as essence for photo browsing through personal digital libraries, Proceedings of the 2nd ACM/IEEE-CS joint conference on Digital libraries, July 14-18, 2002, Portland, Oregon, USA [doi>10.1145/544220.544301]
• 16: O. Marques and N. Barman. "Semi-automatic Semantic Annotation of Images Using Machine Learning Techniques." In The Semantic Web - ISWC 2003 Proceedings, 2003, pp. 550--565.
• 17: B Shevade, H Sundaram, Min Yen-Kan. A Collaborative Annotation Framework. ICME 2005, pp. 1346--1349
• 18: WordNet: An Electronic Lexical Database. Christiane Fellbaum (editor). 1998. The MIT Press, Cambridge, MA.
• 19: H. Liu , P. Singh, ConceptNet — A Practical Commonsense Reasoning Tool-Kit, BT Technology Journal, v.22 n.4, p.211-226, October 2004 [doi>10.1023/B:BTTJ.0000047600.45421.6d]
• 20: B. Shevade, H. Sundaram, L. Xie. Exploiting Personal And Social Network Context For Event Annotation. ICME 2007, pp. 835--838.
• 21: Hameeda Alshabib , Omer F. Rana , Ali Shaikh Ali, Deriving Ratings Through Social Network Structures, Proceedings of the First International Conference on Availability, Reliability and Security, p.779-787, April 20-22, 2006 [doi>10.1109/ARES.2006.50]
• 22: Ming Zhao, Yong Wei Teo, Siliang Liu, Tat-Seng Chua and Ramesh Jain. Automatic Person Annotation of Family Photo Album. CIVR 2006, LNCS 4071, pp. 163--172, 2006.
• 23: Cameron Marlow , Mor Naaman , Danah Boyd , Marc Davis, HT06, tagging paper, taxonomy, Flickr, academic article, to read, Proceedings of the seventeenth conference on Hypertext and hypermedia, August 22-25, 2006, Odense, Denmark [doi>10.1145/1149941.1149949]
• 24: K. S. Jones and P. Willet, Readings in Information Retrieval, San Francisco: Morgan Kaufmann, 1997.
• 25: Brendan Elliott and Z. Meral Özsoyoglu. A Comparison of Methods for Semantic Photo Annotation Suggestion. ISCIS 2007.
• 26: B. Elliott , S. F. Akgul , Z. M. Ozsoyoglu , E. Manilich, A Framework for Querying Pedigree Data, Proceedings of the 18th International Conference on Scientific and Statistical Database Management, p.71-80, July 03-05, 2006 [doi>10.1109/SSDBM.2006.8]
• 27: D. Beckett (ed.). "RDF/XML Syntax Specification (Revised)." W3C Working Recommendation, February 10, 2004. See http://www.w3.org/TR/rdf-syntax-grammar/.
• 28: Rainer Lienhart and Jochen Maydt. An Extended Set of Haar-like Features for Rapid Object Detection. IEEE ICIP 2002, Vol. 1, pp. 900--903, Sep. 2002.
• 29: Matthew Turk , Alex Pentland, Eigenfaces for recognition, Journal of Cognitive Neuroscience, v.3 n.1, p.71-86, Winter 1991 [doi>10.1162/jocn.1991.3.1.71]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Query formulation

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process
      H.3.5 On-line Information Services
          Subjects: Data sharing

General Terms:
Algorithms, Experimentation, Human Factors, Performance

Keywords:
annotation suggestion, pedigrees, personal photos and video, search languages, semantic multimedia management, social networks