Evaluating cost-sensitive Unsolicited Bulk Email categorization

ABSTRACT

In the recent years, Unsolicited Bulk Email has became an increasingly important problem, with a big economic impact. In this paper, we discuss cost-sensitive Text Categorization methods for UBE filtering. In concrete, we have evaluated a range of Machine Learning methods for the task (C4.5, Naive Bayes, PART, Support Vector Machines and Rocchio), made cost sensitive through several methods (Threshold Optimization, Instance Weighting, and Meta-Cost). We have used the Receiver Operating Characteristic Convex Hull method for the evaluation, that best suits classification problems in which target conditions are not known, as it is the case. Our results do not show a dominant algorithm nor method for making algorithms cost-sensitive, but are the best reported on the test collection used, and approach real-world hand-crafted classifiers accuracy.

REFERENCES

• 1: I. Androutsopoulos, J. Koutsias, K. Chandrinos, G. Paliouras, and C. Spyropoulos. An evaluation of naive bayesian anti-spam filtering. In Proc. of the Work. on Machine Learning in the New Information Age, ECML, 2000.
• 2: Ion Androutsopoulos , John Koutsias , Konstantinos V. Chandrinos , Constantine D. Spyropoulos, An experimental comparison of naive Bayesian and keyword-based anti-spam filtering with personal e-mail messages, Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, p.160-167, July 24-28, 2000, Athens, Greece [doi>10.1145/345508.345569]
• 3: I. Androutsopoulos, G. Paliouras, V. Karkaletsis, G. Sakkis, C. Spyropoulos, and P. Stamatopoulos. Learning to filter spam e-mail: A comparison of a naive bayesian and a memory-based approach. In Proc. of the Work. on Machine Learning and Textual Inf. Access, PKDD, 2000.
• 4: X. Carreras and L. Márquez. Boosting trees for anti-spam email filtering. In Proc. of RANLP, 2001.
• 5: Lorrie Faith Cranor , Brian A. LaMacchia, Spam!, Communications of the ACM, v.41 n.8, p.74-83, Aug. 1998 [doi>10.1145/280324.280336]
• 6: Pedro Domingos, MetaCost: a general method for making classifiers cost-sensitive, Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining, p.155-164, August 15-18, 1999, San Diego, California, United States [doi>10.1145/312129.312220]
• 7: H. Drucker, V. Vapnik, and D. Wu. Automatic text categorization and its applications to text retrieval. IEEE Trans. on Neural Networks, 10(5), 1999.
• 8: eTesting Labs. Brightmail, inc. anti-spam service: Comparative performance test. Technical report, eTesting Labs Inc (Ziff Davis Media Inc.), 2001.
• 9: J. Gómez Hidalgo, M. Maña López, and E. Puertas Sanz. Combining text and heuristics for cost-sensitive spam filtering. In Proc. of CONLL, 2000.
• 10: P. Hoffman and D. Crocker. Unsolicited bulk email: Mechanisms for control. Technical Report UBE-SOL, IMCR-008, Internet Mail Cons., 1998.
• 11: P. Pantel and D. Lin. Spamcop: A spam classification & organization program. In Learning for Text Categorization: Papers from the 1998 Work. AAAI Tech. Rep. WS-98-05, 1998.
• 12: Foster Provost , Tom Fawcett, Robust Classification for Imprecise Environments, Machine Learning, v.42 n.3, p.203-231, March 2001 [doi>10.1023/A:1007601015854]
• 13: M. Sahami, S. Dumais, D. Heckerman, and E. Horvitz. A bayesian approach to filtering junk e-mail. In Learning for Text Categorization: Papers from the 1998 Work. AAAI Tech. Rep. WS-98-05, 1998.
• 14: G. Sakkis, I. Androutsopoulos, G. Paliouras, V. Karkaletsis, C. D. Spyropoulos, and P. Stamatopoulos. Stacking classifiers for anti-spam filtering of e-mail. In Proc. of EMNLP, 2001.
• 15: Fabrizio Sebastiani, Machine learning in automated text categorization, ACM Computing Surveys (CSUR), v.34 n.1, p.1-47, March 2002 [doi>10.1145/505282.505283]
• 16: Kai Ming Ting, Inducing Cost-Sensitive Trees via Instance Weighting, Proceedings of the Second European Symposium on Principles of Data Mining and Knowledge Discovery, p.139-147, September 23-26, 1998
• 17: Ian H. Witten , Eibe Frank, Data mining: practical machine learning tools and techniques with Java implementations, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2000
• 18: Yiming Yang, An Evaluation of Statistical Approaches to Text Categorization, Information Retrieval, v.1 n.1-2, p.69-90, 1999 [doi>10.1023/A:1009982220290]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.5 PATTERN RECOGNITION
      I.5.2 Design Methodology
          Subjects: Classifier design and evaluation

Additional Classification:
  H. Information Systems
  H.4 INFORMATION SYSTEMS APPLICATIONS
      H.4.3 Communications Applications
          Subjects: Electronic mail

  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
      I.2.6 Learning
          Subjects: Induction

General Terms:
Algorithms, Design, Experimentation, Performance

Keywords:
cost-sensitive classification, evaluation, text categorization, unsolicited bulk email