Retrieving lightly annotated images using image similarities

ABSTRACT

Users' search needs are often represented by words and images are retrieved according to such textual queries. Annotation words assigned to the stored images are most useful to connect queries to the images. However, due to annotation cost, quite limited amount of annotation words are available in many cases. When annotations are not given at all, there needs to be some techniques that assign annotations automatically. When only a few annotation words are given to each image (lightly annotated), there need to be some enhancement techniques that best use the available annotations. We address the later problem by estimating word associations to fill in the lexical gap between queries and annotations. The model of word associations can be learned from the data. However, since images are only lightly annotated, their sparseness in computing word associations becomes crucial. To compensate the sparseness, we propose a novel data exploration technique in which image similarities contribute to the estimation of word associations on the assumption that similar images have similar semantic concepts. We experimentally show the potential benefit of our approach.

REFERENCES

• 1: Kobus Barnard , Pinar Duygulu , David Forsyth , Nando de Freitas , David M. Blei , Michael I. Jordan, Matching words and pictures, The Journal of Machine Learning Research, 3, 3/1/2003
• 2: Adam Berger , John Lafferty, Information retrieval as statistical translation, Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, p.222-229, August 15-19, 1999, Berkeley, California, United States [doi>10.1145/312624.312681]
• 3: David M. Blei , Michael I. Jordan, Modeling annotated data, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860460]
• 4: David Elworthy, Retrieval from captioned image databases using natural language processing, Proceedings of the ninth international conference on Information and knowledge management, p.430-437, November 06-11, 2000, McLean, Virginia, United States [doi>10.1145/354756.354850]
• 5: Sharon Flank, A layered approach to NLP-based information retrieval, Proceedings of the 17th international conference on Computational linguistics, August 10-14, 1998, Montreal, Quebec, Canada
• 6: G. W. Furnas , T. K. Landauer , L. M. Gomez , S. T. Dumais, The vocabulary problem in human-system communication, Communications of the ACM, v.30 n.11, p.964-971, Nov. 1987 [doi>10.1145/32206.32212]
• 7: D. Hiemstra. Using Language Models for Information Retrieval. PhD thesis, Centre for Telematics and Information Technology, University of Twente, January 2001.
• 8: M. Inoue. On the need for annotation-based image retrieval. In Workshop on Information Retrieval in Context (IRiX), pages 44--46, Sheffield, UK, July 2004.
• 9: J. Jeon , V. Lavrenko , R. Manmatha, Automatic image annotation and retrieval using cross-media relevance models, Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, July 28-August 01, 2003, Toronto, Canada [doi>10.1145/860435.860459]
• 10: George A. Miller, WordNet: a lexical database for English, Communications of the ACM, v.38 n.11, p.39-41, Nov. 1995 [doi>10.1145/219717.219748]
• 11: Jay M. Ponte , W. Bruce Croft, A language modeling approach to information retrieval, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.275-281, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291008]
• 12: Hinrich Schütze , Jan O. Pedersen, A cooccurrence-based thesaurus and two applications to information retrieval, Information Processing and Management: an International Journal, v.33 n.3, p.307-318, May 1997 [doi>10.1016/S0306-4573(96)00068-4]
• 13: Stan Sclaroff , Marco La Cascia , Saratendu Sethi, Unifying textual and visual cues for content-based image retrieval on the World Wide Web, Computer Vision and Image Understanding, v.75 n.1-2, p.86-98, July/Aug. 1999 [doi>10.1006/cviu.1999.0765]
• 14: Alan F. Smeaton , Ian Quigley, Experiments on using semantic distances between words in image caption retrieval, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.174-180, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243261]
• 15: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972]
• 16: Chengxiang Zhai , John Lafferty, A study of smoothing methods for language models applied to information retrieval, ACM Transactions on Information Systems (TOIS), v.22 n.2, p.179-214, April 2004 [doi>10.1145/984321.984322]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models

General Terms:
Algorithms

Keywords:
annotation-based retrieval, image retrieval, lightly annotated images, word associations