Yet Another Solution to Monitor/Trap SYSCALL with Virtualization Technology (x86)

This article describes an idea to monitor SYSCALL with x86 Hardware Virtualization Technology. It doesn't require visible guest code/data modification, so that in some cases, even Kernel Patch Protection (e.g. Windows x64 PatchGuard) cannot detect it. Here the SYSCALL could also be SYSENTER or INT 80h for system calls.

Actually this solution can be extended to other resource monitoring, so at the end of this article I will give an overview on how to extend this as a generic solution (in a new post, LINK?).


In x86/Intel, syscall (Fast System Call) instruction is invoked by a user application at privilege level 0 to call an OS system-call handler at privilege level 0. In ARM architecture, supervisor Call (SVC, formerly SWI) does the similar thing to request privileged operations or access to system resources from an operating system. 

To be more specific, x86/Intel syscall does so by loading RIP from the IA32_LSTAR MSR(after saving the address of the instruction following SYSCALL, returning RIP, into RCX, so when returning back from ring 0, sysret can load that user mode RIP from RCX, and continue to execute program). The WRMSR instruction ensures that the IA32_LSTAR MSR always contain a canonical address (A #GP will be triggered if an address is not canonical when WRMSR to IA32_LSTAR).


The memory address in IA32_LSTAR MSR is an entry point of kernel system-call handler, it can only be configured by software software (OS kernel, normally), and cannot be a non-canonical address. 

However, the key point is here: what if that address saved in IA32_LSTAR is an invalid canonical memory address (or Non-eXecute memory address). When this happens, a CPU Page Fault (#PF) will be triggered at that address. And the error code indicates that this is a page-not-present instruction fetch exception (For simplicity, we don't consider setting Non-eXecute memory address), and CR2 control register content is just that pre-set invalid canonical memory address. 

Hence, whenever a syscall is invoked by application, an intended page-not-present instruction fetch #PF will be triggered. Then this exception is normally handled by #PF (vector = 14) handler specified in OS IDT. 

In a x86/Intel virtualization environment, page faults (exception vector = 14) can be configured to trigger a VMEXIT. And even more, we can selectively make only certain type of #PFs generate VMexit by configuring the VMCS page-fault error-code mask and page-fault error-code match. For example, only page-not-present instruction fetch #PF can generate a VMexit, any other #PF exceptions (like read/write access to invalid or disallowed memory) won't generate VMexit, instead, they are handled in guest IDT #PF handler normally. By doing so, the performance impact will be minimized. 

So, to summarize this solution, we could do it like this to monitor every syscall invoked by user application without guest OS changes:

1. VMM software traps any write access to IA32_LSTAR MSR, whenever a WRMSR to IA32_LSTAR happens, VMM records the original MSR value that points to the real entry point address of kernel system-call handler, and replace it with a MAGIC & INVALID memory address.
     
2. VMM software configures relevant VMCS structures to cause only page-not-present instruction fetch #PF trigger a VMexit.
    
3. At runtime, whenever such a VMexit type happens, VMM software checks guest CR2 value, if it is equal to the predefined MAGIC and INVALID value, then it means this is an intended #PF exception VMexit (not considering malicious (in)direct call to that MAGIC address), we should discard this #PF, and directly resume guest OS back with a new RIP value (it is just the original MSR value that points to the real entry point address of kernel system-call handler).
   
   Otherwise, if this #PF VMexit indicates that it is an ordinary #PF, VMM software injects this exception event back to guest OS without doing anything, then it will be normally handled by #PF handler in guest IDT table.

Therefore, in this way, whenever a syscall is invoked in user mode application, VMM software will get an notification.

But there is a problem here. You can see that we don't need to change any guest OS kernel, however, the Kernel Patch Protection module (like PatchGuard) will probably detect this by reading MSR IA32_LSTAR and comparing it with the original value. It is very easy to solve this issue by monitoring RDMSR to IA32_LSTAR register, and hiding the real value with a return of the original value previously configured by OS kernel. 


Actually, there is another solution that can also works, see my previous blog on Debug Register usages... by enabling Debug breakpoint on the original address stored in IA32_LSTAR MSR by OS kernel. 


As an aforementioned generic solution to monitor/trap a specific event that we're interested in, here it is:

1. Attempt to change guest software for the purpose of making some certain instruction execution generate an intended exception. (Here it is the syscall execution in this post),
2. Then the virtualization software, VMM or Hypervisor, monitors that intended exception by configuring the corresponding VMCS data structures (e.g. Exception-Bitmap VMCS). 

I will write a new post to explain this solution in greater details later.