Intelligent file scoring system for malware detection from the gray list

ABSTRACT

Currently, the most significant line of defense against malware is anti-virus products which focus on authenticating valid software from a white list, blocking invalid software from a black list, and running any unknown software (i.e., the gray list) in a controlled manner. The gray list, containing unknown software programs which could be either normal or malicious, is usually authenticated or rejected manually by virus analysts. Unfortunately, along with the development of the malware writing techniques, the number of file samples in the gray list that need to be analyzed by virus analysts on a daily basis is constantly increasing. In this paper, we develop an intelligent file scoring system (IFSS for short) for malware detection from the gray list by an ensemble of heterogeneous base-level classifiers derived by different learning methods, using different feature representations on dynamic training sets. To the best of our knowledge, this is the first work of applying such ensemble methods for malware detection. IFSS makes it practical for virus analysts to identify malware samples from the huge gray list and improves the detection ability of anti-virus software. It has already been incorporated into the scanning tool of Kingsoft's Anti-Virus software. The case studies on large and real daily collection of the gray list illustrate that the detection ability and efficiency of our IFSS system outperforms other popular scanning tools such as NOD32 and Kaspersky.

REFERENCES

• 1: Rakesh Agrawal , Tomasz Imieliński , Arun Swami, Mining association rules between sets of items in large databases, Proceedings of the 1993 ACM SIGMOD international conference on Management of data, p.207-216, May 25-28, 1993, Washington, D.C., United States
• 2: Aleksander Kołcz , Xiaomei Sun , Jugal Kalita, Efficient handling of high-dimensional feature spaces by randomized classifier ensembles, Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, July 23-26, 2002, Edmonton, Alberta, Canada [doi>10.1145/775047.775093]
• 3: Elena Baralis , Paolo Garza, A Lazy Approach to Pruning Classification Rules, Proceedings of the 2002 IEEE International Conference on Data Mining, p.35, December 09-12, 2002
• 4: Constraint-Based Rule Mining in Large, Dense Databases, Proceedings of the 15th International Conference on Data Engineering, p.188, March 23-26, 1999
• 5: U. Bayer, A. Moser, C. Kruegel, and E. Kirda. Dynamic analysis of malicious code. J Comput Virol, 2:67--77, May 2006.
• 6: N.V. Chawla, K.W. Bowyer, L.O. Hall, and W.P. Kegelmeyer. SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16:321--357, 2002.
• 7: Mihai Christodorescu , Somesh Jha , Christopher Kruegel, Mining specifications of malicious behavior, Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, September 03-07, 2007, Dubrovnik, Croatia [doi>10.1145/1287624.1287628]
• 8: Thomas G. Dietterich. Machine-learning research: Four current directions. AI Magazine, 18(4):97--136, 1997.
• 9: Thomas G. Dietterich, Ensemble Methods in Machine Learning, Proceedings of the First International Workshop on Multiple Classifier Systems, p.1-15, June 21-23, 2000
• 10: D.K.S.Reddy and A.K.Pujari. N-gram analysis for computer virus detection. J Comput Virol, 2:231--239, November 2006.
• 11: Saso Džeroski , Bernard Ženko, Is Combining Classifiers with Stacking Better than Selecting the Best One?, Machine Learning, v.54 n.3, p.255-273, March 2004 [doi>10.1023/B:MACH.0000015881.36452.6e]
• 12: Eric Filiol, Gregoire Jacob, and Michael Le Liard. Evaluation methodology and theoretical model for antiviral behavioural detection strategies. Journal in Computer Virology, 3(1):27--37, 2007.
• 13: Jiawei Han , Jian Pei , Yiwen Yin, Mining frequent patterns without candidate generation, Proceedings of the 2000 ACM SIGMOD international conference on Management of data, p.1-12, May 15-18, 2000, Dallas, Texas, United States
• 14: C. Hsu and C. Lin. A comparison of methods for multiclass support vector machines. IEEE Trans. Neural Networks, 13:415--425, 2002.
• 15: Hwanjo Yu , Jiong Yang , Jiawei Han, Classifying large data sets using SVMs with hierarchical clusters, Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, August 24-27, 2003, Washington, D.C. [doi>10.1145/956750.956786]
• 16: Ivor W. Tsang , James T. Kwok , Pak-Ming Cheung, Core Vector Machines: Fast SVM Training on Very Large Data Sets, The Journal of Machine Learning Research, 6, p.363-392, 12/1/2005
• 17: Anil K. Jain , Robert P. W. Duin , Jianchang Mao, Statistical Pattern Recognition: A Review, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.22 n.1, p.4-37, January 2000 [doi>10.1109/34.824819]
• 18: Nathalie Japkowicz , Shaju Stephen, The class imbalance problem: A systematic study, Intelligent Data Analysis, v.6 n.5, p.429-449, October 2002
• 19: Jeremy Z. Kolter , Marcus A. Maloof, Learning to detect malicious executables in the wild, Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, August 22-25, 2004, Seattle, WA, USA [doi>10.1145/1014052.1014105]
• 20: P. Langley. Selection of relevant features in machine learning. In Proceedings of AAAI Fall Symp., 1994.
• 21: Wenmin Li , Jiawei Han , Jian Pei, CMAR: Accurate and Efficient Classification Based on Multiple Class-Association Rules, Proceedings of the 2001 IEEE International Conference on Data Mining, p.369-376, November 29-December 02, 2001
• 22: B. Liu, W. Hsu, and Y. Ma. Integrating classification and association rule mining. In Proceedings of KDD'98, pages 80--86, 1998.
• 23: Manish Mehta , Rakesh Agrawal , Jorma Rissanen, SLIQ: A Fast Scalable Classifier for Data Mining, Proceedings of the 5th International Conference on Extending Database Technology: Advances in Database Technology, p.18-32, March 25-29, 1996
• 24: Christopher J. Merz, Using Correspondence Analysis to Combine Classifiers, Machine Learning, v.36 n.1-2, p.33-58, July-August 1999 [doi>10.1023/A:1007559205422]
• 25: Hanchuan Peng , Fuhui Long , Chris Ding, Feature Selection Based on Mutual Information: Criteria of Max-Dependency, Max-Relevance, and Min-Redundancy, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.8, p.1226-1238, August 2005 [doi>10.1109/TPAMI.2005.159]
• 26: Matthew G. Schultz , Eleazar Eskin , Erez Zadok , Salvatore J. Stolfo, Data Mining Methods for Detection of New Malicious Executables, Proceedings of the 2001 IEEE Symposium on Security and Privacy, p.38, May 14-16, 2001
• 27: A. H. Sung , J. Xu , P. Chavez , S. Mukkamala, Static Analyzer of Vicious Executables (SAVE), Proceedings of the 20th Annual Computer Security Applications Conference, p.326-334, December 06-10, 2004 [doi>10.1109/CSAC.2004.37]
• 28: G.J. Tesauro, J.O. Kephart, and G.B. Sorkin. Neural networks for computer virus recognition. IEEE Expert, 11:5--6, 1996.
• 29: FADI THABTAH, A review of associative classification mining, The Knowledge Engineering Review, v.22 n.1, p.37-65, March 2007 [doi>10.1017/S0269888907001026]
• 30: J. Wang, P. Deng, Y. Fan, L. Jaw, and Y. Liu. Virus detection using data mining techniques. In Proceedings ICDM'03, 2003.
• 31: Y. Wang, Q. Xin, and F. Coenen. A novel rule ordering approach in classification association rule mining. In Proceedings of ICDM Workshop, 2007.
• 32: David H. Wolpert, Stacked generalization, Neural Networks, v.5 n.2, p.241-259, 1992 [doi>10.1016/S0893-6080(05)80023-1]
• 33: Yanfang Ye , Dingding Wang , Tao Li , Dongyi Ye, IMDS: intelligent malware detection system, Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, August 12-15, 2007, San Jose, California, USA [doi>10.1145/1281192.1281308]
• 34: X. Yin and J. Han. CPAR: Classification based on predictive association rules. In Proceedings of SIAM International Conference on Data Mining (SDM-03), pages 331--335, 2003.

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.6 Security and Protection
          Subjects: Invasive software (e.g., viruses, worms, Trojan horses)

General Terms:
Algorithms, Experimentation, Security

Keywords:
ensemble, gray list, malware detection