Learning-based linguistic indexing of pictures with 2--d MHMMs

ABSTRACT

Automatic linguistic indexing of pictures is an important but highly challenging problem for researchers in computer vision and content-based image retrieval. In this paper, we introduce a statistical modeling approach to this problem. Categorized images are used to train a dictionary of hundreds of concepts automatically based on statistical modeling. Images of any given concept category are regarded as instances of a stochastic process that characterizes the category. To measure the extent of association between an image and the textual description of a category of images, the likelihood of the occurrence of the image based on the stochastic process derived from the category is computed. A high likelihood indicates a strong association. In our experimental implementation, the ALIP (Automatic Linguistic Indexing of Pictures) system, we focus on a particular group of stochastic processes for describing images, that is, the two-dimensional multiresolution hidden Markov models (2-D MHMMs). We implemented and tested the system on a photographic image database of 600 different semantic cat- egories, each with about 40 training images. Tested using 3,000 images outside the training database, the system has demonstrated good accuracy and high potential in linguistic indexing of these test images.

REFERENCES

• 1: K. Barnard, D. Forsyth, "Learning the Semantics of Words and Pictures," Proc. ICCV, vol 2, pp. 408--415, 2001.
• 2: Christopher M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, Inc., New York, NY, 1995
• 3: L. Breiman, J. Friedman, C.J. Stone, R.A. Olshen, Classification and Regression Trees, Wadsworth Int. Group, Belmont, Calif., 1984.
• 4: Chad Carson , Megan Thomas , Serge Belongie , Joseph M. Hellerstein , Jitendra Malik, Blobworld: A System for Region-Based Image Indexing and Retrieval, Proceedings of the Third International Conference on Visual Information and Information Systems, p.509-516, June 02-04, 1999
• 5: R. Chellappa and A. K. Jain, Markov Random Fields: Theory and Applications, Academic Press, 1993.
• 6: Yixin Chen , James Z. Wang, A Region-Based Fuzzy Feature Matching Approach to Content-Based Image Retrieval, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.9, p.1252-1267, September 2002 [doi>10.1109/TPAMI.2002.1033216]
• 7: Wesley W. Chu , Ion T. Ieong , Ricky K. Taira, A semantic modeling approach for image retrieval by content, The VLDB Journal — The International Journal on Very Large Data Bases, v.3 n.4, October 1994
• 8: Nello Cristianini , John Shawe-Taylor, An introduction to support Vector Machines: and other kernel-based learning methods, Cambridge University Press, New York, NY, 1999
• 9: Ingrid Daubechies, Ten lectures on wavelets, Society for Industrial and Applied Mathematics, Philadelphia, PA, 1992
• 10: R. L. Dobrushin, "The description of a random field by means of conditional probabilities and conditions of its regularity," Theory Prob. Appl., vol. 13, pp. 197--224, 1968.
• 11: Myron Flickner , Harpreet Sawhney , Wayne Niblack , Jonathan Ashley , Qian Huang , Byron Dom , Monika Gorkani , Jim Hafner , Denis Lee , Dragutin Petkovic , David Steele , Peter Yanker, Query by Image and Video Content: The QBIC System, Computer, v.28 n.9, p.23-32, September 1995 [doi>10.1109/2.410146]
• 12: David A. Forsyth , Jean Ponce, Computer Vision: A Modern Approach, Prentice Hall Professional Technical Reference, 2002
• 13: S. Geman and D. Geman, "Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. PAMI-6, no. 6, pp. 721--741, Nov. 1984.
• 14: R. Kindermann and L. Snell, Markov Random Fields and Their Applications, American Mathematical Society, 1980.
• 15: J. Li, R. M. Gray, R. A. Olshen, "Multiresolution image classification by hierarchical modeling with two dimensional hidden Markov models," IEEE Trans. on Information Theory, vol. 46, no. 5, pp. 1826--41, August 2000.
• 16: Jia Li , Robert M. Gray, Image Segmentation and Compression Using Hidden Markov Models, Kluwer Academic Publishers, Norwell, MA, 2000
• 17: J. Li, A. Najmi, R. M. Gray, "Image classification by a two dimensional hidden Markov model," IEEE Trans. on Signal Processing, vol. 48, no. 2, pp. 517--33, February 2000.
• 18: W. Y. Ma , B. S. Manjunath, NeTra: a toolbox for navigating large image databases, Proceedings of the 1997 International Conference on Image Processing (ICIP '97) 3-Volume Set-Volume 1, p.568, October 26-29, 1997
• 19: Y. C. Park , P. K. Kim , F. Golshani , S. Panchanathan, Technique for eliminating irrelevant terms in term rewriting for annotated media retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500247]
• 20: A. Pentland, R. W. Picard, S. Sclaroff, "Photobook: Tools for content-based manipulation of image databases," Proc. of SPIE, vol. 2185, pp. 34--47, San Jose, February 7--8, 1994.
• 21: Jianbo Shi , Jitendra Malik, Normalized Cuts and Image Segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.8, p.888-905, August 2000 [doi>10.1109/34.868688]
• 22: J. R. Smith, S.-F. Chang, "An image and video search engine for the World-Wide Web," Proc. of SPIE, vol. 3022, pp. 84--95, 1997.
• 23: M. Unser, "Texture classification and segmentation using wavelet frames," IEEE Trans. Image Processing, vol. 4, no. 11, pp. 1549--1560, Nov. 1995.
• 24: James Z. Wang, Integrated Region-Based Image Retrieval, Kluwer Academic Publishers, Norwell, MA, 2001
• 25: James Z. Wang , Jia Li , Gio Wiederhold, SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture LIbraries, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.9, p.947-963, September 2001 [doi>10.1109/34.955109]
• 26: James Z. Wang , Jia Li , Robert M. Gray , Gio Wiederhold, Unsupervised Multiresolution Segmentation for Images with Low Depth of Field, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.1, p.85-90, January 2001 [doi>10.1109/34.899949]
• 27: Song Chun Zhu , Alan Yuille, Region Competition: Unifying Snakes, Region Growing, and Bayes/MDL for Multiband Image Segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.18 n.9, p.884-900, September 1996 [doi>10.1109/34.537343]

INDEX TERMS

Keywords:
computer vision, content-based image retrieval, hidden Markov model, image classification, image segmentation, machine learning, region matching, wavelets