Putting active learning into multimedia applications: dynamic definition and refinement of concept classifiers

ABSTRACT

The authors developed an extensible system for video exploitation that puts the user in control to better accommodate novel situations and source material. Visually dense displays of thumbnail imagery in storyboard views are used for shot-based video exploration and retrieval. The user can identify a need for a class of audiovisual detection, adeptly and fluently supply training material for that class, and iteratively evaluate and improve the resulting automatic classification produced via multiple modality active learning and SVM. By iteratively reviewing the output of the classifier and updating the positive and negative training samples with less effort than typical for relevance feedback systems, the user can play an active role in directing the classification process while still needing to truth only a very small percentage of the multimedia data set. Examples are given illustrating the iterative creation of a classifier for a concept of interest to be included in subsequent investigations, and for a concept typically deemed irrelevant to be weeded out in follow-up queries. Filtering and browsing tools making use of existing and iteratively added concepts put the user further in control of the multimedia browsing and retrieval process.

REFERENCES

• 1: Christopher Ahlberg , Ben Shneiderman, Visual information seeking: tight coupling of dynamic query filters with starfield displays, Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, p.313-317, April 24-28, 1994, Boston, Massachusetts, United States [doi>10.1145/191666.191775]
• 2: Christopher J. C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, v.2 n.2, p.121-167, June 1998 [doi>10.1023/A:1009715923555]
• 3: Chang, E.Y., Tong, S., and Goh, K.-S. Support Vector Machine Concept-Dependent Active Learning for Image Retrieval. IEEE Transactions on Multimedia (anticipated 2005), http://mmdb2.ece.ucsb.edu/~echang/mm000540.pdf.
• 4: Shih-Fu Chang , Gwendal Auffret , Jonathan Foote , Chung-Shen Li , Behzad Shahraray , Tanveer Syeda-Mahmood , HongJiang Zhang, Multimedia access and retrieval (panel session): the state of the art and future directions, Proceedings of the seventh ACM international conference on Multimedia (Part 1), p.443-445, October 30-November 05, 1999, Orlando, Florida, United States [doi>10.1145/319463.319684]
• 5: Michael G. Christel , Ronald M. Conescu, Addressing the challenge of visual information access from digital image and video libraries, Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2005, Denver, CO, USA [doi>10.1145/1065385.1065402]
• 6: David A. Forsyth , Jean Ponce, Computer Vision: A Modern Approach, Prentice Hall Professional Technical Reference, 2002
• 7: Yoav Freund , Robert E. Schapire, A decision-theoretic generalization of on-line learning and an application to boosting, Journal of Computer and System Sciences, v.55 n.1, p.119-139, Aug. 1997 [doi>10.1006/jcss.1997.1504]
• 8: Gosselin, P.H., and Cord, M. RETIN AL: An active learning strategy for image category retrieval. In Proc. IEEE Conf. Image Processing (Singapore, October 2004), 2219--2222.
• 9: Hastie, T., Tibshirani, R., and Friedman, J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2001.
• 10: Alexander G. Hauptmann , Michael G. Christel, Successful approaches in the TREC video retrieval evaluations, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027681]
• 11: Lee, H. and Smeaton, A.F. Designing the User Interface for the Fischlar Digital Video Library, J. Digital Info. 2(4), http://jodi.ecs.soton.ac.uk/Articles/v02/i04/Lee/, May 2002.
• 12: Andrew McCallum , Kamal Nigam, Employing EM and Pool-Based Active Learning for Text Classification, Proceedings of the Fifteenth International Conference on Machine Learning, p.350-358, July 24-27, 1998
• 13: Naphade, M., and Smith, J.R. Active Learning for Simultaneous Annotation of Multiple Binary Concepts. In Proc. IEEE Intl. Conf. on Multimedia and Expo (ICME) (Taipei, Taiwan, June, 2004), 77--80.
• 14: Milind R. Naphade , John R. Smith, On the detection of semantic concepts at TRECVID, Proceedings of the 12th annual ACM international conference on Multimedia, October 10-16, 2004, New York, NY, USA [doi>10.1145/1027527.1027680]
• 15: Hieu T. Nguyen , Arnold Smeulders, Active learning using pre-clustering, Proceedings of the twenty-first international conference on Machine learning, p.79, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015349]
• 16: Rowe, L.A. and Jain, R., ACM SIGMM Retreat Report on Future Directions in Multimedia Research, http://www.sigmm.org/Events/reports/retreat03/sigmm-retreat03-final.pdf, March, 2004.
• 17: H. Schneiderman , T. Kanade, Probabilistic Modeling of Local Appearance and Spatial Relationships for Object Recognition, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, p.45, June 23-25, 1998
• 18: Simon Tong , Edward Chang, Support vector machine active learning for image retrieval, Proceedings of the ninth ACM international conference on Multimedia, September 30-October 05, 2001, Ottawa, Canada [doi>10.1145/500141.500159]
• 19: Trant, J. Image Retrieval Benchmark Database Service: A Needs Assessment and Preliminary Develoment Plan. Council on Library and Information Resources and the Coalition for Networked Information, Archives & Museum Informatics, http://www.clir.org/pubs/reports/ trant04/tranttext.pdf, January 2004.
• 20: Wang, L., Chan, K.L., and Zhang, Z. Bootstrapping SVM Active Learning by Incorporating Unlabelled Images for Image Retrieval. In Conf. Computer Vision and Pattern Recognition (CVPR '03) (Madison, WI, June, 1998). IEEE Computer Society, 2003, 629--634.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems
          Subjects: Video (e.g., tape, disk, DVI)

General Terms:
Algorithms, Experimentation, Human Factors

Keywords:
active learning, extensible concept classification, video retrieval