Traps, events, emulation, and enforcement: managing the yin and yang of virtualization-based security

ABSTRACT

We question current trends that attempt to leverage virtualization techniques to achieve security goals. We suggest that the security role of a virtual machine centers on being a policy interpreter rather than a resource provider. These two roles (security reference monitor and resource emulator) are currently conflated within the context of virtual machines and VMMs. We believe that this ``double-duty'' leads to both a significant performance impact as well as a bloated virtualization layer. Increased complexity reduces confidence that the code is elementary enough to verify or trust from a security perspective. Ironically, as more security-related functionality is shoved into a VM platform, the system becomes less trustworthy as it becomes increasingly trusted.

We argue that a principle reason for such an unfortunate situation is the lack of efficient hardware trapping mechanisms. We propose an architecture to help ameliorate this problem by transferring the security enforcement and program analysis roles from the virtualization component to a policy-directed FPGA.

REFERENCES

• 1: Martín Abadi , Mihai Budiu , Úlfar Erlingsson , Jay Ligatti, Control-flow integrity, Proceedings of the 12th ACM conference on Computer and communications security, November 07-11, 2005, Alexandria, VA, USA [doi>10.1145/1102120.1102165]
• 2: H. Agrawal. Towards Automatic Debugging of Computer Programs, August 1991.
• 3: T. Beauchamp and D. Weston. Re:Trace - Applied Reverse Engineering on OS X.
• 4: T. Beauchamp and D. Weston. Re:Trace - Applied Reverse Engineering on OS X. RECON 2008, 2008. Montreal, Quebec.
• 5: Fabrice Bellard, QEMU, a fast and portable dynamic translator, Proceedings of the annual conference on USENIX Annual Technical Conference, p.41-41, April 10-15, 2005, Anaheim, CA
• 6: Steven M. Bellovin, Virtual machines, virtual security?, Communications of the ACM, v.49 n.10, October 2006 [doi>10.1145/1164394.1164414]
• 7: Frederick P. Brooks, Jr., The mythical man-month (anniversary ed.), Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1995
• 8: Derek Bruening , Timothy Garnett , Saman Amarasinghe, An infrastructure for adaptive dynamic optimization, Proceedings of the international symposium on Code generation and optimization: feedback-directed and runtime optimization, March 23-26, 2003, San Francisco, California
• 9: Bryan Buck , Jeffrey K. Hollingsworth, An API for Runtime Code Patching, International Journal of High Performance Computing Applications, v.14 n.4, p.317-329, November 2000 [doi>10.1177/109434200001400404]
• 10: Bryan M. Cantrill , Michael W. Shapiro , Adam H. Leventhal, Dynamic instrumentation of production systems, Proceedings of the annual conference on USENIX Annual Technical Conference, p.2-2, June 27-July 02, 2004, Boston, MA
• 11: P. A. Karger and D. R. Safford. Security and Performance Trade-Offs in I/O Operations for Virtual Machine Monitors. In IBM Research Technical Report RC24500 (W0802--069), February 2008.
• 12: Samuel T. King , Joseph Tucek , Anthony Cozzie , Chris Grier , Weihang Jiang , Yuanyuan Zhou, Designing and implementing malicious hardware, Proceedings of the 1st Usenix Workshop on Large-Scale Exploits and Emergent Threats, p.1-8, April 15-15, 2008, San Francisco, California
• 13: Vladimir Kiriansky , Derek Bruening , Saman P. Amarasinghe, Secure Execution via Program Shepherding, Proceedings of the 11th USENIX Security Symposium, p.191-206, August 05-09, 2002
• 14: Chi-Keung Luk , Robert Cohn , Robert Muth , Harish Patil , Artur Klauser , Geoff Lowney , Steven Wallace , Vijay Janapa Reddi , Kim Hazelwood, Pin: building customized program analysis tools with dynamic instrumentation, Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, June 12-15, 2005, Chicago, IL, USA
• 15: mayhem. The Cerberus ELF Interface. Phrack, 2003.
• 16: Nicholas Nethercote , Julian Seward, Valgrind: a framework for heavyweight dynamic binary instrumentation, Proceedings of the 2007 ACM SIGPLAN conference on Programming language design and implementation, June 10-13, 2007, San Diego, California, USA
• 17: Edmund B. Nightingale , Daniel Peek , Peter M. Chen , Jason Flinn, Parallelizing security checks on commodity hardware, Proceedings of the 13th international conference on Architectural support for programming languages and operating systems, March 01-05, 2008, Seattle, WA, USA
• 18: Jeffrey Oplinger , Monica S. Lam, Enhancing software reliability with speculative threads, Proceedings of the 10th international conference on Architectural support for programming languages and operating systems, October 05-09, 2002, San Jose, California
• 19: V. Prasad, W. Cohen, F. C. Eigler, M. Hunt, J. Keniston, and B. Chen. Locating system problems using dynamic instrumentation. 2005.
• 20: Timothy Roscoe , Kevin Elphinstone , Gernot Heiser, Hype and virtue, Proceedings of the 11th USENIX workshop on Hot topics in operating systems, p.1-6, May 07-09, 2007, San Diego, CA
• 21: Emre C. Sezer , Peng Ning , Chongkyung Kil , Jun Xu, Memsherlock: an automated debugger for unknown memory corruption vulnerabilities, Proceedings of the 14th ACM conference on Computer and communications security, October 28-31, 2007, Alexandria, Virginia, USA [doi>10.1145/1315245.1315314]
• 22: T. E. shell crew. Embedded ELF Debugging: the middle head of Cerberus. Phrack, 2003.
• 23: R. M. Stallman, R. H. Pesch, and S. Shebs. Debugging with GDB: The GNU Source-Level Debugger. Free Software Foundation, 2003.
• 24: H. Yin, Z. Liang, and D. Song. HookFinder: Identifying and Understanding Malware Hooking Behaviors. In Proceedings of the 15th Annual Network and Distributed System Security Symposium (NDSS), February 2008.

INDEX TERMS

Primary Classification:
  B. Hardware
  B.7 INTEGRATED CIRCUITS
      B.7.1 Types and Design Styles
          Subjects: Gate arrays

Additional Classification:
  C. Computer Systems Organization
  C.1 PROCESSOR ARCHITECTURES
      C.1.3 Other Architecture Styles
          Subjects: Data-flow architectures

General Terms:
Design, Performance, Security

Keywords:
debugging, security policy, traps, virtualization