Evaluating strategies and systems for content based indexing of person images on the Web

ABSTRACT

Content based indexing of multimedia has always been a challenging task. The enormity and the diversity of the multimedia content on the web adds another dimension to this challenge. In this paper, we examine ways of combining visual and textual information for content based indexing of multimedia on the web. In particular, we examine different methods of combining evidences due to face detection, Text/HTML analysis and face recognition for identifying person images. We provide experimental evaluation of the following strategies: i) Face detection on the image followed by Text/HTML analysis of the containing page; ii) face detection followed by face recognition; iii) face detection followed by a linear combination of evidences due to text/HTML analysis and face recognition; and iv) face detection followed by a Dempster-Shafer combination of evidences due to text/HTML analysis and face recognition. These strategies were implemented in an automatic web search agent named Diogenes¹ and compared against some well known web image search engines. The latter includes commercial systems such as Alta Vista, Lycos and Ditto, and a research prototype, WebSEEk. We report the results of our experimental retrievals where Diogenes outperformed these search engines for celebrity image queries in terms of average precision.

REFERENCES

• 1: Y. Alp Aslandogan , Chuck Thier , Clement Yu, A system for effective content based image retrieval, Proceedings of the fourth ACM international conference on Multimedia, p.429-430, November 18-22, 1996, Boston, Massachusetts, United States [doi>10.1145/244130.244455]
• 2: Y. Alp Aslandogan , Chuck Thier , Clement T. Yu , Jon Zou , Naphtali Rishe, Using semantic contents and WordNet in image retrieval, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, p.286-295, July 27-31, 1997, Philadelphia, Pennsylvania, United States
• 3: Y. Alp Aslandogan , Clement T. Yu, Multiple evidence combination in image retrieval: Diogenes searches for people on the Web, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.88-95, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345553]
• 4: Y. Alp Aslandogan and Clement Yu. Multiple Evidence Combination in Image retrieval: Diogenes Searches for People on the Web. Technical Report UIC-EECS-00-1, Department of EECS, University of Illinois at Chicago, January 2000.
• 5: Y. Alp Aslandogan , Clement T. Yu, Techniques and Systems for Image and Video Retrieval, IEEE Transactions on Knowledge and Data Engineering, v.11 n.1, p.56-63, January 1999 [doi>10.1109/69.755615]
• 6: N. J. Belkin , P. Kantor , E. A. Fox , J. A. Shaw, Combining the evidence of multiple query representations for information retrieval, Information Processing and Management: an International Journal, v.31 n.3, p.431-448, May-June 1995
• 7: Chua, T., Pung, H., Lu, G., and Jong, H. A Concept Based Image Retrieval System. In IEEE Int'nl Conf. on system Sciences, pages 590-598, January 1994.
• 8: C. Faloutsos , R. Barber , M. Flickner , J. Hafner , W. Niblack , D. Petkovic , W. Equitz, Efficient and effective querying by image content, Journal of Intelligent Information Systems, v.3 n.3-4, p.231-262, July 1994 [doi>10.1007/BF00962238]
• 9: Theo Gevers and Arnold W. M. Smenlders. PicToSeek: A Content-Based Image Search System for the World Wide Web. In Proceedings of SPIE Visual 97, 1997.
• 10: David Lee Hall, Mathematical Techniques in Multisensor Data Fusion, Artech House, Inc., Norwood, MA, 1992
• 11: Joemon M. Jose , Jonathan Furner , David J. Harper, Spatial querying for image retrieval: a user-oriented evaluation, Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, p.232-240, August 24-28, 1998, Melbourne, Australia [doi>10.1145/290941.291000]
• 12: M. La Cascia , S. Sethi , S. Sclaroff, Combining Textual and Visual Cues for Content-Based Image Retrieval on the World Wide Web, Proceedings of the IEEE Workshop on Content - Based Access of Image and Video Libraries, p.24, June 21-21, 1998
• 13: Joon Ho Lee, Analyses of multiple evidence combination, Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, p.267-276, July 27-31, 1997, Philadelphia, Pennsylvania, United States
• 14: Sougata Mukherjea , Kyoji Hirata , Yoshinori Hara, AMORE: A World Wide Web image retrieval engine, World Wide Web, v.2 n.3, p.115-132, 1999 [doi>10.1023/A:1019248722478]
• 15: Olaf Munkelt, Oliver Kaufmann, and Wolfgang Eckstein. Content-based Image Retrieval in the World Wide Web: A Web Agent for Fetching Portraits. In Proceedings of SPIE Vol. 3022, pages 408-416, 1997.
• 16: Henry A. Rowley , Shumeet Baluja , Takeo Kanade, Neural Network-Based Face Detection, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.20 n.1, p.23-38, January 1998 [doi>10.1109/34.655647]
• 17: Gerald Salton, Automatic text processing, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1988
• 18: Ashok Samal , Prasana A. Iyengar, Automatic recognition and analysis of human faces and facial expressions: a survey, Pattern Recognition, v.25 n.1, p.65-77, Jan. 1992 [doi>10.1016/0031-3203(92)90007-6]
• 19: Glenn Shafer. A Mathematical Theory of Evidence. Princeton University Press, 1976.
• 20: A. Prasad Sistla , Clement T. Yu , R. Haddad, Reasoning About Spatial Relationships in Picture Retrieval Systems, Proceedings of the 20th International Conference on Very Large Data Bases, p.570-581, September 12-15, 1994
• 21: Alan F. Smeaton , Ian Quigley, Experiments on using semantic distances between words in image caption retrieval, Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, p.174-180, August 18-22, 1996, Zurich, Switzerland [doi>10.1145/243199.243261]
• 22: John R. Smith , Shih-Fu Chang, Visually Searching the Web for Content, IEEE MultiMedia, v.4 n.3, p.12-20, July 1997 [doi>10.1109/93.621578]
• 23: Charles Frankel , Michael J Swain , Vassilis Athitsos, WebSeer: An Image Search Engine for the World Wide Web, University of Chicago, Chicago, IL, 1996
• 24: Leonid Taycher, Marco LaCaseia, and Stan Selaroff. Image Digestion and Relevance Feedback in the ImageRover WWW Search Engine. In Proceedings of SPIB Visual 97, 1997.
• 25: M. Turk and A. Pentland. Eigenfaces for Recognition. Cognitive Neuroseience, 3(1):71-86, 1991.
• 26: Clement T. Yu , Weiyi Meng, Principles of database query processing for advanced applications, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1998
• 27: L. A. Zadeh, A Simple view of the Dempster-Shafer theory of evidence and its, University of California at Berkeley, Berkeley, CA, 1985

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.8 Database applications
          Subjects: Image databases
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Retrieval models
      H.3.5 On-line Information Services
          Subjects: Web-based services
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.3 Group and Organization Interfaces
          Subjects: Web-based interaction

  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Feature evaluation and selection

General Terms:
Design, Experimentation, Human Factors, Management, Measurement, Performance, Theory

Keywords:
Dempster-Shafer theory, content based image retrieval, evidence combination, face detection and recognition, web image retrieval