Using engine signature to detect metamorphic malware

ABSTRACT

This paper introduces the "engine signature" approach to assist in detecting metamorphic malware by tracking it to its engine. More specifically, it presents and evaluates a code scoring technique for collecting forensic evidence from x86 code segments in order to get some measure of how likely they are to have been generated by some known instruction-substituting metamorphic engine. A prototype simulator that mimics real instruction-substituting metamorphic engines was implemented and used to conduct several experiments that evaluate the goodness of the scoring technique for given engine parameters. The technique was also used to successfully help track variants of W32.Evol to their engine.

REFERENCES

• 1: Brushi, D., Martignoni, L., and Monga, M. Using Code Normalization for Fighting Self-Mutating Malware. In Proceedings of the International Symposium of Secure Software Engineering (Arlington, VA, 2006).
• 2: Chess, D. M., and White, S. R. An Undetectable Computer Virus. In Proceedings of Virus Bulletin Conference (2000).
• 3: Cohen, F. Computational Aspects of Computer Viruses. Computers & Security, 8 (1989), 325--344.
• 4: Mihai Christodorescu , Somesh Jha , Sanjit A. Seshia , Dawn Song , Randal E. Bryant, Semantics-Aware Malware Detection, Proceedings of the 2005 IEEE Symposium on Security and Privacy, p.32-46, May 08-11, 2005 [doi>10.1109/SP.2005.20]
• 5: Karim, M. E., Walenstein, A., Lakhotia, A., and Parida, L. Malware Phylogeny Generation using Permutations of Code. European Research Journal of Computer Virology 1, 1-2 (Nov. 2005) 13--23.
• 6: Krsul, I., Spafford, E. H. Authorship Analysis: Identifying The Author of a Program. Technical Report 96--052, 1996.
• 7: Kruegel, C., Kirda, E., Mutz, D., Robertson, W., and Vigna, G. Polymorphic Worm Detection Using Structural Information of Executables. In Proceedings of the 8 th Symposium on Recent Advances in Intrusion Detection (RAID) (Seattle, WA, USA, September 7-9, 2005).
• 8: Spinellis, D. Reliable Identification of Bounded-Length Viruses is NP-Complete. IEEE Transactions on Information Theory, 49, 1 (2003), 280--284.
• 9: Peter Szor, The Art of Computer Virus Research and Defense, Addison-Wesley Professional, 2005
• 10: Andrew Walenstein , Rachit Mathur , Mohamed R. Chouchane , Arun Lakhotia, Normalizing Metamorphic Malware Using Term Rewriting, Proceedings of the Sixth IEEE International Workshop on Source Code Analysis and Manipulation (SCAM'06), p.75-84, September 27-29, 2006 [doi>10.1109/SCAM.2006.20]

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS
      D.4.6 Security and Protection
          Subjects: Invasive software (e.g., viruses, worms, Trojan horses)

Additional Classification:
  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.5 Security and Protection (D.4.6, K.4.2)
          Subjects: Invasive software (e.g., viruses, worms, Trojan horses)

General Terms:
Experimentation, Measurement, Security

Keywords:
metamorphic engine, virus scanner