Vidya: an experiential annotation system

ABSTRACT

In this paper, we present a novel annotation paradigm with an emphasis on two facets -- (a) the end user experience and (b) semantic propagation. The annotation problem is important since media semantics play a key role in new multimedia applications. However, there is currently very little incentive for end users to annotate.The annotation system, is interactive and experiential. We attempt to propagate semantics of the annotations, by using WordNet, a lexicographic arrangement of words, and low-level features extracted from the images. We introduce novel semantic dissimilarity measures, and propagation frameworks. The system provides insight to the user, by providing her with knowledge sources, that are constrained by the user and media context. The knowledge sources are presented using context-aware hyper-mediation.Our Experimental results indicates that the systems performs well. We tested the new annotation experience using a pilot user study, the users agreed that the new framework was more useful that a traditional annotation interface. The semantic propagation results are good as well -- we converge close to the semantics of the image by annotating a small number (~15%) of database images.

REFERENCES

• 1: Merriam Webster Dictionary <u>http://www.m-w.com.</u>
• 2: Google <u>http://www.google.com.</u>
• 3: OpenCyc <u>http://www.opencyc.org.</u>
• 4: V. BUSH (1945). As We May Think. <u>The Atlantic Monthly</u>. 176: 101--108, <u>http://www.theatlantic.com/unbound/flashbks/computer/bushf.htm.</u>
• 5: E. CHANG, K. GOH, G. SYCHAY, et al. (2003). CBSA: content-based soft annotation for multimodal image retrieval using Bayes point machines. <u>IEEE Transactions on Circuits and Systems for Video Technology</u> 13(1): 26--38.
• 6: Ingemar J. Cox , Joumana Ghosh , Matt L. Miller , Thomas V. Papathomas , Peter N. Yianilos, Hidden Annotation in Content Based Image Retrieval, Proceedings of the 1997 Workshop on Content-Based Access of Image and Video Libraries (CBAIVL '97), p.76, June 20-20, 1997
• 7: Chitra Dorai , Andreas Mauthe , Frank Nack , Lloyd Rutledge , Thomas Sikora , Herbert Zettl, Media semantics: who needs it and why?, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641123]
• 8: Xiaofei He , Wei-Ying Ma , Oliver King , Mingjing Li , Hongjiang Zhang, Learning and inferring a semantic space from user's relevance feedback for image retrieval, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641080]
• 9: Anil K. Jain, Fundamentals of digital image processing, Prentice-Hall, Inc., Upper Saddle River, NJ, 1989
• 10: Ramesh Jain, Experiential computing, Communications of the ACM, v.46 n.7, July 2003 [doi>10.1145/792704.792729]
• 11: Ramesh Jain, Folk computing, Communications of the ACM, v.46 n.4, April 2003 [doi>10.1145/641205.641223]
• 12: I. KANT, P. GUYER and A. W. WOOD (1998). <u>Critique of pure reason</u>. Cambridge ; New York, Cambridge University Press.
• 13: G. A. MILLER, R. BECKWITH and C. FELLBAUMIntroduction to WordNet: An on-Line Lexical Database. <u>International Journal of Lexicography</u> 3(4): 235--244.
• 14: G. A. MILLER, R. BECKWITH and C. FELLBAUM (1993). Introduction to WordNet: An on-Line Lexical Database. <u>International Journal of Lexicography</u> 3(4): 235--244.
• 15: Theodore H. Nelson, Computer Lib/Dream Machines, Microsoft Press, Redmond, WA, 1987
• 16: J. RAMAMOORTHY, S. WONG and H. SUNDARAM (2003). Dynamic Adaptive Visualizations. Arts Media and Engineering Center, ASU, AME-TR-2003-03, 2003.
• 17: Yong Rui , Thomas S. Huang, A novel relevance feedback technique in image retrieval, Proceedings of the seventh ACM international conference on Multimedia (Part 2), p.67-70, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319878.319896]
• 18: Direct Annotation: A Drag-and-Drop Strategy for Labeling Photos, Proceedings of the International Conference on Information Visualisation, p.88, July 19-21, 2000
• 19: H. SRIDHARAN, H. SUNDARAM and T. RIKAKIS (2003). Context, memory and Hyper-mediation in Experiential Systems. Arts Media and Engineering Center, ASU, AME-TR-2003-02, 2003.
• 20: Yanfeng Sun , Hongjiang Zhang , Lei Zhang , Mingjing Li, MyPhotos: a system for home photo management and processing, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641022]
• 21: H. SUNDARAM and T. RIKAKIS (2003). An Introduction to Experiential Systems. Arts Media and Engineering Center, ASU, AME-2003-01, 2003.
• 22: L. WENYIN, S. DUMAIS, Y. SEN, et al. (2001). Semi-Automatic Image Annotation, Proc. Human-Computer Interaction--Interact 01, pp. 326--333,
• 23: Ye Lu , Chunhui Hu , Xingquan Zhu , HongJiang Zhang , Qiang Yang, A unified framework for semantics and feature based relevance feedback in image retrieval systems, Proceedings of the eighth ACM international conference on Multimedia, p.31-37, October 2000, Marina del Rey, California, United States [doi>10.1145/354384.354403]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.4 Hypertext/Hypermedia
          Subjects: Navigation

General Terms:
Algorithms, Experimentation, Human Factors, Languages, Theory

Keywords:
annotation, experiential, hyper-mediation, semantic propagation, serendipity