Hierarchical classification for automatic image annotation

ABSTRACT

In this paper, a hierarchical classification framework has been proposed for bridging the semantic gap effectively and achieving multi-level image annotation automatically. First, the semantic gap between the low-level computable visual features and users' real information needs is partitioned into four smaller gaps, and multiple approachesallare proposed to bridge these smaller gaps more effectively. To learn more reliable contextual relationships between the atomic image concepts and the co-appearances of salient objects, a multi-modal boosting algorithm is proposed. To enable hierarchical image classification and avoid inter-level error transmission, a hierarchical boosting algorithm is proposed by incorporating concept ontology and multi-task learning to achieve hierarchical image classifier training with automatic error recovery. To bridge the gap between the computable image concepts and the users' real information needs, a novel hyperbolic visualization framework is seamlessly incorporated to enable intuitive query specification and evaluation by acquainting the users with a good global view of large-scale image collections. Our experiments on large-scale image databases have also obtained very positive results.

REFERENCES

• 1: Arnold W. M. Smeulders , Marcel Worring , Simone Santini , Amarnath Gupta , Ramesh Jain, Content-Based Image Retrieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.12, p.1349-1380, December 2000 [doi>10.1109/34.895972]
• 2: Xin-Jing Wang , Lei Zhang , Feng Jing , Wei-Ying Ma, AnnoSearch: Image Auto-Annotation by Search, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.1483-1490, June 17-22, 2006 [doi>10.1109/CVPR.2006.58]
• 3: W. Y. Ma , B. S. Manjunath, Texture Features and Learning Similarity, Proceedings of the 1996 Conference on Computer Vision and Pattern Recognition (CVPR '96), p.425, June 18-20, 1996
• 4: J. Jeon, R. Manmatha, "Using maximum entropy for automatic image annotation", CIVR, pp. 24-32, 2004.
• 5: Jianping Fan , Yuli Gao , Hangzai Luo , Guangyou Xu, Automatic image annotation by using concept-sensitive salient objects for image content representation, Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, July 25-29, 2004, Sheffield, United Kingdom [doi>10.1145/1008992.1009055]
• 6: Y. Freund, R. E. Schapire, "Experiments with a new boosting algorithm", Proc. ICML, pp. 148-156, 1996.
• 7: Robert E. Schapire , Yoram Singer, BoosTexter: A Boosting-based Systemfor Text Categorization, Machine Learning, v.39 n.2-3, p.135-168, May-June 2000 [doi>10.1023/A:1007649029923]
• 8: A. Torralba, K. P. Murphy, W. T. Freeman, "Sharing features: efficient boosting procedures for multiclass object detection", IEEE CVPR, 2004.
• 9: K. Yu, A. Schwaighofor, V. Tresp, W. -Y. Ma, H. J. Zhang, "Collaborative ensemble learning: Combining content-based information filtering via hierarchical Bayes", Proc. of Intl. Conf. on Uncertainty in Artificial Intelligence (UAI), 2003.
• 10: Theodoros Evgeniou , Charles A. Micchelli , Massimiliano Pontil, Learning Multiple Tasks with Kernel Methods, The Journal of Machine Learning Research, 6, p.615-637, 12/1/2005
• 11: George A. Miller, WordNet: a lexical database for English, Communications of the ACM, v.38 n.11, p.39-41, Nov. 1995 [doi>10.1145/219717.219748]
• 12: S. Deerwester, S. T. Dumais and R. Harshman, "Indexing by latent semantic analysis", Journal of the American Society of Information Science 1990.
• 13: Yossi Rubner , Carlo Tomasi , Leonidas J. Guibas, A Metric for Distributions with Applications to Image Databases, Proceedings of the Sixth International Conference on Computer Vision, p.59, January 04-07, 1998
• 14: J. A. Walter, D. Webling, K. Essig, H. Ritter, "Interactive hyperbolic image browsing-towards an integrated multimedia navigator", KDD, 2006.
• 15: G. P. Nguyen, M. Worring, "Similarity based visualization of image collections", AVIVDiLib, 2005.
• 16: J. Lamping, R. Rao, "The hyperbolic browser: A focus + content technique for visualizing large hierarchies", Journal of Visual Languages and Computing 1996.
• 17: Donald Ervin Knuth, The Art of Computer Programming, 2nd Ed. (Addison-Wesley Series in Computer Science and Information, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1978
• 18: J. Friedman, T. Hastie, R. Tibshirani, "Additive logistic regression: a statistical view of boosting", Annals of Statistics vol. 28, no. 2, pp. 337--374, 2000.
• 19: K. Grauman, T. Darrell, "The Pyramid Match Kernel: Discriminative Classification with Sets of Image Features", MIT-CSAIL-TR-2006-20, MIT 2006.
• 20: Jianping Fan , Hangzai Luo , Mohand-Said Hacid, Mining images on semantics via statistical learning, Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, August 21-24, 2005, Chicago, Illinois, USA [doi>10.1145/1081870.1081877]
• 21: K. Barnard and D. Forsyth, "Learning the semantics of words and pictures", IEEE ICCV, 2001.

INDEX TERMS

Primary Classification:
  F. Theory of Computation
  F.m MISCELLANEOUS

General Terms:
Algorithms

Keywords:
concept ontology, hierarchical boosting, image classification, multi-task learning, semantic gap