Hybrid analysis of executables to detect security vulnerabilities: security vulnerabilities

ABSTRACT

Detection of vulnerabilities in executables is one of the major challenges facing the software industry and is mainly due to the unavailability of the source code. In this work, we present a hybrid approach which is a combination of static and dynamic analysis to identify vulnerabilities. In this approach, we first instrument the executable using PIN to extract the control flow and the corresponding assembly code using disassembler. We then perform static analysis on the assembly code for constraint bound checking using control flow and register bounds. In this way, we exploit the synergy between static and dynamic analysis to detect memory leaks, buffer overflow and dangling pointers.

REFERENCES

• 1: J. Bergeron , Mourad Debbabi , M. M. Erhioui , Béchir Ktari, Static Analysis of Binary Code to Isolate Malicious Behaviors, Proceedings of the 8th Workshop on Enabling Technologies on Infrastructure for Collaborative Enterprises, p.184-189, June 16-18, 1999
• 2: M. D. Ernst. Static and dynamic analysis: synergy and duality. In In WODA 2003: ICSE Workshop on Dynamic Analysis, pages 24--27, 2003.
• 3: Chi-Keung Luk , Robert Cohn , Robert Muth , Harish Patil , Artur Klauser , Geoff Lowney , Steven Wallace , Vijay Janapa Reddi , Kim Hazelwood, Pin: building customized program analysis tools with dynamic instrumentation, Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, June 12-15, 2005, Chicago, IL, USA

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.5 Security and Protection (D.4.6, K.4.2)
          Subjects: Invasive software (e.g., viruses, worms, Trojan horses)

Additional Classification:
  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.5 Security and Protection (D.4.6, K.4.2)
          Subjects: Unauthorized access (e.g., hacking, phreaking)

General Terms:
Algorithms, Design, Experimentation, Security

Keywords:
hybrid analysis, instrumentation, memory errors, security vulnerabilities, slicing