Improve Performance for Separating Kernel and User Address Space with Process-Context Identifiers (PCIDs)

This post is not talking about any new idea, just about what I'm thinking..

Back to year 2003, Ingo Molnar proposed 4G/4G split on x86 with 64 GB RAM support to separate user and kernel mode virtual address space. This is another post 64GB on 32-bit systems talking about this. Originally, the motivation was as below in that post. 

The 4G/4G split feature is primarily intended for large-RAM x86 systems, which want to (or have to) get more kernel/user VM, at the expense of per-syscall TLB-flush overhead.

Obviously this is true for 32bit OS, for example in Linux OS, by default the kernel uses higher 1GB virtual address, while user space uses lower 3GB virtual space. In Windows OS, by default 2G/2G split is used.

At that time, security design was not a big concern, however, in year 2010 (maybe before this date),  PaX team revisited this again by [grsec] Announcing UDEREF/amd64. But this time the motivation is not about kernel/user VM size, but about securing kernel and user address space to mitigate many ret2user attacks. 

Before hardware PCID (Process-Context Identifiers) feature was introduced, TLB flush due to user/kernel page table switch through syscall or interrupt/exception will cause significant performance cost. In latest UDEREF/amd64 implementation (see this link: http://grsecurity.net/stable/grsecurity-3.0-3.14.24-201411150026.patch, PaX team told me that they will blog this), this PCID hardware feature will be used to speed up performance. 


So what is PCID?  see text below from Intel SDM.

Process-context identifiers (PCIDs) are a facility by which a logical processor may cache information for multiple linear-address spaces. The processor may retain cached information when software switches to a different linear address space with a different PCID.

When a logical processor creates entries in the TLBs and paging-structure caches, it associates those entries with the current PCID. When using entries in the TLBs and paging-structure caches to translate a linear address, a logical processor uses only those entries associated with the current PCID.

However, the PCID feature is only available on x64 mode (Intel IA-32e mode), which means only 64bit OS can use it. The PCID is a 12-bit value stored in CR3 register for each address space, see below from SDM manual. 

To assisting in OS software programing, the new instruction INVPCID (Invalidate Process-Context Identifier) also is introduced to invalidate mappings in the translation lookaside buffers (TLBs) and paging-structure caches based on process context identifier (PCID). It is kind of like INVEPT and INVVPID in Intel Virtualization technology, the former is to invalidate information cached from the EPT paging structures, and the later is to invalidates mappings in the translation lookaside buffers (TLBs) and paging-structure caches based on Virtual Processor Identifier (VPID). 

There are four INVPCID types (granularities) currently defined (copied from IA32/Intel SDM):

1. Individual-address invalidation: If the INVPCID type is 0, the logical processor invalidates mappings—except global translations—for the linear address and PCID specified in the INVPCID descriptor. In some cases, the instruction may invalidate global translations or mappings for other linear addresses (or other PCIDs) as well.
2. Single-context invalidation: If the INVPCID type is 1, the logical processor invalidates all mappings—except global translations—associated with the PCID specified in the INVPCID descriptor. In some cases, the instruction may invalidate global translations or mappings for other PCIDs as well.
3. All-context invalidation, including global translations: If the INVPCID type is 2, the logical processor invalidates all mappings—including global translations—associated with any PCID.
4. All-context invalidation: If the INVPCID type is 3, the logical processor invalidates all mappings—except global translations—associated with any PCID. In some case, the instruction may invalidate global translations as well.

References:
IA32 Intel Software Development Manual... just searching it in Google..

4G/4G split on x86, 64 GB RAM (and more) support
http://lwn.net/Articles/39283/

64GB on 32-bit systems
http://lwn.net/Articles/39925/

[grsec] Announcing UDEREF/amd64
http://grsecurity.net/pipermail/grsecurity/2010-April/001024.html

Grsecurity patch download
http://grsecurity.net/download.php