Real-time computerized annotation of pictures

ABSTRACT

Automated annotation of digital pictures has been a highly challenging problem for computer scientists since the invention of computers. The capability of annotating pictures by computers can lead to breakthroughs in a wide range of applications including Web image search, online picture-sharing communities, and scientific experiments. In our work, by advancing statistical modeling and optimization techniques, we can train computers about hundreds of semantic concepts using example pictures from each concept. The ALIPR (Automatic Linguistic Indexing of Pictures -Real Time)system of fully automatic and high speed annotation for online pictures has been constructed. Thousands of pictures from an Internet photo-sharing site, unrelated to the source of those pictures used in the training process, have been tested. The experimental results show that a single computer processor can suggest annotation terms in real-time and with good accuracy.

REFERENCES

• 1: Kobus Barnard , Pinar Duygulu , David Forsyth , Nando de Freitas , David M. Blei , Michael I. Jordan, Matching words and pictures, The Journal of Machine Learning Research, 3, 3/1/2003
• 2: D. Beymer and T. Poggio, "Image representations for visual learning," Science vol. 272, pp. 1905--1909, 1996.
• 3: P.J. Bickel and D.A. Freedman, "Some asymptotic theory for the bootstrap," Annals of Statistics vol. 9, pp. 1196--1217, 1981.
• 4: S.-F. Chang, W .Chen, and H. Sundaram, "Semantic visual templates: Linking visual features to semantics," In Proc. Int. Conf. on Image Processing, vol. 3, pp. 531--535, Chicago, IL, 1998.
• 5: Ritendra Datta , Jia Li , James Z. Wang, Content-based image retrieval: approaches and trends of the new age, Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval, November 10-11, 2005, Hilton, Singapore [doi>10.1145/1101826.1101866]
• 6: Ingrid Daubechies, Ten lectures on wavelets, Society for Industrial and Applied Mathematics, Philadelphia, PA, 1992
• 7: M. Evans, N. Hastings, and B. Peacock, Statistical Distributions 3rd ed., John Wiley & Sons, Inc., 2000.
• 8: Rafael C. Gonzalez , Richard E. Woods, Digital Image Processing, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2001
• 9: T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inferences, and Prediction Springer-Verlag, New York, 2001.
• 10: Jingrui He , Hanghang Tong , Mingjing Li , Hong-Jiang Zhang , Changshui Zhang, Mean version space: a new active learning method for content-based image retrieval, Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval, October 15-16, 2004, New York, NY, USA [doi>10.1145/1026711.1026715]
• 11: Xiaofei He , Wei-Ying Ma , Hong-Jiang Zhang, Learning an image manifold for retrieval, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027532]
• 12: E. Levina and P. Bickel, "The earth mover's distance is the Mallows distance: Some insights from statistics," In Proc. Int. Conf. on Computer Vision pp. 251--256, Vancouver, Canada, 2001.
• 13: Jia Li , James Z. Wang, Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.25 n.9, p.1075-1088, September 2003 [doi>10.1109/TPAMI.2003.1227984]
• 14: C.L. Mallows, "A not on asymptotic joint normality," Annals of Mathematical Statistics vol. 43, no. 2, pp. 508--515, 1972.
• 15: Florent Monay , Daniel Gatica-Perez, On image auto-annotation with latent space models, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957070]
• 16: Y. Rui, T.S. Huang, M. Ortega, and S. Mehrotra, "Relevance feedback: A power tool in interactive content-based image retrieval," IEEE Transactions on Circuits and Systems for Video Technology, vol. 8, no. 5, pp. 644--655, 1998.
• 17: A. Singhal, J. Luo, and W. Zhu, "Probabilistic spatial context models for scene content understanding," In Proc. IEEE Int. Conf. on Computer Vision and Pattern Recognition 2003.
• 18: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972]
• 19: John R. Smith , Shih-Fu Chang, VisualSEEk: a fully automated content-based image query system, Proceedings of the fourth ACM international conference on Multimedia, p.87-98, November 18-22, 1996, Boston, Massachusetts, United States [doi>10.1145/244130.244151]
• 20: Kinh Tieu , Paul Viola, Boosting Image Retrieval, International Journal of Computer Vision, v.56 n.1-2, p.17-36, January-February 2004 [doi>10.1023/B:VISI.0000004830.93820.78]
• 21: C. Tomasi, "Past performance and future results," Nature vol. 428, page 378, March 2004.
• 22: Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500159]
• 23: N. Vasconcelos and A. Lippman, "A multiresolution manifold distance for invariant image similarity," IEEE Transactions on Multimedia vol. 7, no. 1, pp. 127--142, 2005.
• 24: James Z. Wang , Jia Li , Gio Wiederhold, SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture LIbraries, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.9, p.947-963, September 2001 [doi>10.1109/34.955109]
• 25: James Z. Wang , Jia Li, Learning-based linguistic indexing of pictures with 2--d MHMMs, Proceedings of the tenth ACM international conference on Multimedia, December 01-06, 2002, Juan-les-Pins, France [doi>10.1145/641007.641104]
• 26: C. Zhang and T. Chen, "An active learning framework for content-based information retrieval," IEEE Transactions on Multimedia vol. 4, no. 2, pp. 260--268, 2002.

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.3 Clustering
          Subjects: Algorithms

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval

General Terms:
Algorithms, Experimentation, Performance

Keywords:
clustering, image annotation, modeling, statistical learning