Virtual Machine-based program modification

Background

Virtualization techniques has made it possible to insert extra software layers underneath software that was previously running at the lowest system level, such as OS kernels.

The extra software layer have been used to multiplex the system resources into isolated virtual machines, migrating running systems from one physical host to another, etc.

But the possibilities do not end there.  

Objectives

To investigate the potential of Virtual Machine-based program modification for run-time security monitoring of binary programs, the first objective is to develop a software instrumentation toolkit that can run inside a virtual machine monitor.   The toolkit will enable the virtual machine monitor to do on-the-fly modifications to the executing machine code as it is running. This will enable enforcement of arbitrary fine grained security properties of the running programs.

The second objective is to develop a framework security policies, including a language, and means to compile the policies into instrumentation instructions for the instrumentation toolkit, so that the policy is applied to the running code.

Task description

The toolkit will be built using the LLVM software for the internal representation of the code, and will use just-in-time compilation techniques from dynamically typed languages, including specialization, inlining, etc., to achieve efficient dynamic code generation.

One task is to implement emulation code for the machine instructions of the Intel x86 family processors, including support for the more complex instructions that change the GDT, IDT, and CRs, as the virtual machines will run system level code.

Another task is to investigate what kind of security checks that can efficiently be added to the code, and show by a concrete example how to encode them into a language, that is parsed and executed from the toolkit developed in the first task. 

The two tasks have to be carried out concurrently, as the requirements from the second task influences what interface the toolkit should provide to the programmer.