Efficient discriminative learning of Bayesian network classifier via boosted augmented naive Bayes

ABSTRACT

The use of Bayesian networks for classification problems has received significant recent attention. Although computationally efficient, the standard maximum likelihood learning method tends to be suboptimal due to the mismatch between its optimization criteria (data likelihood) and the actual goal for classification (label prediction). Recent approaches to optimizing the classification performance during parameter or structure learning show promise, but lack the favorable computational properties of maximum likelihood learning. In this paper we present the Boosted Augmented Naive Bayes (BAN) classifier. We show that a combination of discriminative data-weighting with generative training of intermediate models can yield a computationally efficient method for discriminative parameter learning and structure selection.

REFERENCES

• 1: Blake, C., & Merz, C. (1998). UCI repository of machine learning databases.
• 2: Chow, C. K., & Liu, C. N. (1968). Approximating discrete probability distributions with dependence trees. IEEE Transactions on Information Theory, IT-14, 462--467.
• 3: Gregory F. Cooper , Edward Herskovits, A Bayesian Method for the Induction of Probabilistic Networks from Data, Machine Learning, v.9 n.4, p.309-347, Oct. 1992 [doi>10.1023/A:1022649401552]
• 4: Elkan, C. (1997). Boosting and naive bayesian learning (Technical Report). Department of Computer Science and Engineering, University of California, San Diego.
• 5: Friedman, J., Hastie, T., & Tibshirani, R. (2000). Additive logistic regression: a statistical view of boosting. The Annals of Statistics, 38, 337--374.
• 6: Nir Friedman , Dan Geiger , Moises Goldszmidt, Bayesian Network Classifiers, Machine Learning, v.29 n.2-3, p.131-163, Nov./Dec. 1997 [doi>10.1023/A:1007465528199]
• 7: Russell Greiner , Wei Zhou, Structural extension to logistic regression: discriminative parameter learning of belief net classifiers, Eighteenth national conference on Artificial intelligence, p.167-173, July 28-August 01, 2002, Edmonton, Alberta, Canada
• 8: Daniel Grossman , Pedro Domingos, Learning Bayesian network classifiers by maximizing conditional likelihood, Proceedings of the twenty-first international conference on Machine learning, p.46, July 04-08, 2004, Banff, Alberta, Canada [doi>10.1145/1015330.1015339]
• 9: Heckerman, D. (1999). A tutorial on learning with bayesian networks. 301--354.
• 10: Ng, A. Y., & Jordan, M. I. (2002). On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. Proc. 14th Conference on Advances in Neural Information Processing Systems (pp. 841--848). Cambridge, MA: MIT Press.
• 11: Ridgeway, G., Madigan, D., Richardson, T., & O'Kane, J. (1998). Interpretable boosted naive bayes classification. Proceedings Fourth International Conference on Knowledge Discovery and Data Mining.
• 12: Robert E. Schapire , Yoram Singer, Improved boosting algorithms using confidence-rated predictions, Proceedings of the eleventh annual conference on Computational learning theory, p.80-91, July 24-26, 1998, Madison, Wisconsin, United States [doi>10.1145/279943.279960]
• 13: Geoffrey I. Webb , Zijian Zheng, Multistrategy Ensemble Learning: Reducing Error by Combining Ensemble Learning Techniques, IEEE Transactions on Knowledge and Data Engineering, v.16 n.8, p.980-991, August 2004 [doi>10.1109/TKDE.2004.29]
• 14: Geoffrey I. Webb , Janice R. Boughton , Zhihai Wang, Not So Naive Bayes: Aggregating One-Dependence Estimators, Machine Learning, v.58 n.1, p.5-24, January 2005 [doi>10.1007/s10994-005-4258-6]