Diverse ensembles for active learning

ABSTRACT

Query by Committee is an effective approach to selective sampling in which disagreement amongst an ensemble of hypotheses is used to select data for labeling. Query by Bagging and Query by Boosting are two practical implementations of this approach that use Bagging and Boosting, respectively, to build the committees. For effective active learning, it is critical that the committee be made up of consistent hypotheses that are very different from each other. DECORATE is a recently developed method that directly constructs such diverse committees using artificial training data. This paper introduces ACTIVE-DECORATE, which uses DECORATE committees to select good training examples. Extensive experimental results demonstrate that, in general, ACTIVE-DECORATE outperforms both Query by Bagging and Query by Boosting.

REFERENCES

• 1: Naoki Abe , Hiroshi Mamitsuka, Query Learning Strategies Using Boosting and Bagging, Proceedings of the Fifteenth International Conference on Machine Learning, p.1-9, July 24-27, 1998
• 2: Blake, C. L., & Merz, C. J. (1998). UCI repository of machine learning databases. http://www.ics.uci.edu/~mlearn/MLRepository.html.
• 3: David Cohn , Les Atlas , Richard Ladner, Improving Generalization with Active Learning, Machine Learning, v.15 n.2, p.201-221, May 1994 [doi>10.1023/A:1022673506211]
• 4: Cohn, D. A., Ghahramani, Z., & Jordan, M. I. (1996). Active learning with statistical models. Journal of Artificial Intelligence Research, 4, 129--145.
• 5: Thomas M. Cover , Joy A. Thomas, Elements of information theory, Wiley-Interscience, New York, NY, 1991
• 6: Dagan, I., & Engelson, S. P. (1995). Committee-based sampling for training probabilistic classifiers. Proc. of 12th Intl. Conf. on Machine Learning (ICML-95) (pp. 150--157). San Francisco, CA: Morgan Kaufmann.
• 7: Yoav Freund , H. Sebastian Seung , Eli Shamir , Naftali Tishby, Selective Sampling Using the Query by Committee Algorithm, Machine Learning, v.28 n.2-3, p.133-168, Aug./Sept. 1997 [doi>10.1023/A:1007330508534]
• 8: Juan Francisco Gómez-Lopera , José Martínez-Aroza , Aureliano M. Robles-Pérez , Ramón Román-Roldán, An Analysis of Edge Detection by Using the Jensen-Shannon Divergence, Journal of Mathematical Imaging and Vision, v.13 n.1, p.35-56, August 2000 [doi>10.1023/A:1008325607354]
• 9: Krogh, A., & Vedelsby, J. (1995). Neural network ensembles, cross validation and active learning. Advances in Neural Information Processing Systems 7.
• 10: Lewis, D. D., & Catlett, J. (1994). Heterogeneous uncertainty sampling for supervised learning. Proc. of 11th Intl. Conf. on Machine Learning (ICML-94) (pp. 148--156). San Francisco, CA: Morgan Kaufmann.
• 11: Liere, R., & Tadepalli, P. (1997). Active learning with committees for text categorization. Proc. of 14th Natl. Conf. on Artificial Intelligence (AAAI-97) (pp. 591--596). Providence, RI.
• 12: Andrew McCallum , Kamal Nigam, Employing EM and Pool-Based Active Learning for Text Classification, Proceedings of the Fifteenth International Conference on Machine Learning, p.350-358, July 24-27, 1998
• 13: Melville, P., & Mooney, R. (2003). Constructing diverse classifier ensembles using artificial training examples. Proc. of 18th Intl. Joint Conf. on Artificial Intelligence (pp. 505--510). Acapulco, Mexico.
• 14: Melville, P., & Mooney, R. J. (2004). Creating diversity in ensembles using artificial data. Information Fusion: Special Issue on Diversity in Multiclassifier Systems.
• 15: Melville, P., Saar-Tsechansky, M., Provost, F., & Mooney, R. (2004). Active feature acquisition for classifier induction. Submitted for review. Available at http://www.cs.utexas.edu/users/ml/publication/.
• 16: Ion Muslea , Steven Minton , Craig A. Knoblock, Selective Sampling with Redundant Views, Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, p.621-626, July 30-August 03, 2000
• 17: J. Ross Quinlan, C4.5: programs for machine learning, Morgan Kaufmann Publishers Inc., San Francisco, CA, 1993
• 18: Nicholas Roy , Andrew McCallum, Toward Optimal Active Learning through Sampling Estimation of Error Reduction, Proceedings of the Eighteenth International Conference on Machine Learning, p.441-448, June 28-July 01, 2001
• 19: Saar-Tsechansky, M., & Provost, F. J. (2001). Active learning for class probability estimation and ranking. Proc. of 17th Intl. Joint Conf. on Artificial Intelligence (IJCAI-2001) (pp. 911--920).
• 20: H. S. Seung , M. Opper , H. Sompolinsky, Query by committee, Proceedings of the fifth annual workshop on Computational learning theory, p.287-294, July 27-29, 1992, Pittsburgh, Pennsylvania, United States [doi>10.1145/130385.130417]
• 21: Ian H. Witten , Eibe Frank, Data mining: practical machine learning tools and techniques with Java implementations, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2000
• 22: Zhu, X., Lafferty, J., & Ghahramani, Z. (2003). Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions. Proc. of the ICML Workshop on the Continuum from Labeled to Unlabeled Data (pp. 58--65).