CVS log for src/sys/arch/x86/x86/svs.c

Up to [cvs.NetBSD.org] / src / sys / arch / x86 / x86

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

x86: Support EFI runtime services.

This creates a special pmap, efi_runtime_pmap, which avoids setting
PTE_U but allows mappings to lie in what would normally be user VM --
this way we don't fall afoul of SMAP/SMEP when executing EFI runtime
services from CPL 0.  SVS does not apply to the EFI runtime pmap.

The mechanism is intended to work with either physical addressing or
virtual addressing; currently the bootloader does physical addressing
but in principle it could be modified to do virtual addressing
instead, if it allocated virtual pages, assigned them in the memory
map, and issued RT->SetVirtualAddressMap.

Not sure pmap_activate_sync and pmap_deactivate_sync are correct,
need more review from an x86 wizard.

If this causes fallout, it can be disabled temporarily without
reverting anything by just making efi_runtime_init return immediately
without doing anything, or by removing options EFI_RUNTIME.

amd64-only for now pending type fixes and testing on i386.

x86: Split most of pmap.h into pmap_private.h or vmparam.h.

This way pmap.h only contains the MD definition of the MI pmap(9)
API, which loads of things in the kernel rely on, so changing x86
pmap internals no longer requires recompiling the entire kernel every
time.

Callers needing these internals must now use machine/pmap_private.h.
Note: This is not x86/pmap_private.h because it contains three parts:

1. CPU-specific (different for i386/amd64) definitions used by...

2. common definitions, including Xenisms like xpmap_ptetomach,
   further used by...

3. more CPU-specific inlines for pmap_pte_* operations

So {amd64,i386}/pmap_private.h defines 1, includes x86/pmap_private.h
for 2, and then defines 3.  Maybe we should split that out into a new
pmap_pte.h to reduce this trouble.

No functional change intended, other than that some .c files must
include machine/pmap_private.h when previously uvm/uvm_pmap.h
polluted the namespace with pmap internals.

Note: This migrates part of i386/pmap.h into i386/vmparam.h --
specifically the parts that are needed for several constants defined
in vmparam.h:

VM_MAXUSER_ADDRESS
VM_MAX_ADDRESS
VM_MAX_KERNEL_ADDRESS
VM_MIN_KERNEL_ADDRESS

Since i386 needs PDP_SIZE in vmparam.h, I added it there on amd64
too, just to keep things parallel.

KNF. No functional change.

Revert most of ad's movs/stos change. Instead do a lot simpler: declare
svs_quad_copy() used by SVS only, with no need for instrumentation, because
SVS is disabled when sanitizers are on.

Introduce per-cpu IDTs

This is realized by following modifications:
- Add IDT pages and its allocation maps for each cpu in "struct cpu_info"
- Load per-cpu IDTs at cpu_init_idt(struct cpu_info*)
- Copy the IDT entries for cpu0 to other CPUs at attach
   - These are, for example, exceptions, db, system calls, etc.

And, added a kernel option named PCPU_IDT to enable the feature.

svs_pdir_switch(): Use MOVS to copy the PTES.

svs_pmap_sync(): Fast-path the curcpu case.  Could be improved further
with a kcpuset iterator thing.

Modify the hotpatch mechanism, in order to make it much less ROP-friendly.

Currently x86_patch_window_open is a big problem, because it is a perfect
function to inject/modify executable code with ROP.

 - Remove x86_patch_window_open(), along with its x86_patch_window_close()
   counterpart.
 - Introduce a read-only link-set of hotpatch descriptor structures,
   which reference a maximum of two read-only hotpatch sources.
 - Modify x86_hotpatch() to open a window and call the new
   x86_hotpatch_apply() function in a hard-coded manner.
 - Modify x86_hotpatch() to take a name and a selector, and have
   x86_hotpatch_apply() resolve the descriptor from the name and the
   source from the selector, before hotpatching.
 - Move the error handling in a separate x86_hotpatch_cleanup() function,
   that gets called after we closed the window.

The resulting implementation is a bit complex and non-obvious. But it
gains the following properties: the code executed in the hotpatch window
is strictly hard-coded (no callback and no possibility to execute your own
code in the window) and the pointers this code accesses are strictly
read-only (no possibility to forge pointers to hotpatch an area that was
not designated as hotpatchable at compile-time, and no possibility to
choose what bytes to write other than the maximum of two read-only
templates that were designated as valid for the given destination at
compile-time).

With current CPUs this slightly improves a situation that is already
pretty bad by definition on x86. Assuming CET however, this change closes
a big hole and is kinda great.

The only ~problem there is, is that dtrace-fbt tries to hotpatch random
places with random bytes, and there is just no way to make it safe.
However dtrace is only in a module, that is rarely used and never compiled
into the kernel, so it's not a big problem; add a shitty & vulnerable
independent hotpatch window in it, and leave big XXXs. It looks like fbt
is going to collapse soon anyway.

Merge the bouyer-xenpvh branch, bringing in Xen PV drivers support under HVM
guests in GENERIC.
Xen support can be disabled at runtime with
boot -c
disable hypervisor

Sync with bouyer-xenpvh-base2 (HEAD)

Give the ldt a fixed size of one page (512 slots), and drop the variable-
sized mechanism that was too complex.

This fixes a race between USER_LDT and SVS: during context switches, the
way SVS installs the new ldt relies on the ldt pointer AND the ldt size,
but both cannot be accessed atomically at the same time.

Mostly merge changes from HEAD upto 20200411

Merge changes from current as of 20200406

Sync with head.

'oldlwp' is never NULL now, so remove the NULL checks.

Merge x86 pmap changes from yamt-pagecache:

- Deal better with the multi-level pmap object locking kludge.
- Handle uvm_pagealloc() being able to block.

Add support for USER_LDT in SVS. This allows us to have both enabled at
the same time.

We allocate an LDT for each CPU in the GDT and map an area for it, in
addition to the default LDT already present. In context switches between
different processes, we choose between the default or the per-cpu LDT
selector: if the user set specific LDT entries, we memcpy them to the
per-cpu LDT and load the per-cpu selector.

Tested by Naveen Narayanan (with Wine on amd64).

Sync with HEAD

Add PCID support in SVS. This avoids TLB flushes during kernel<->user
transitions, which greatly reduces the performance penalty introduced by
SVS.

We use two ASIDs, 0 (kern) and 1 (user), and use invpcid to flush pages
in both ASIDs.

The read-only machdep.svs.pcid={0,1} sysctl is added, and indicates whether
SVS+PCID is in use.

Change the effect of SVS on the TLB. Keep CR4_PGE set when SVS is enabled,
but don't use PTE_G on the kernel PTEs in general.

Add PTE_G on only a few pages, that are already leaked to userland and do
not contain secrets.

This slightly improves syscall performance.

Remove 'ci_svs_kpdirpa', unused. While here fix a few comments here and
there, reduces a future diff.

Change the way SVS is disabled. Now you have to pass "boot -3" from the
bootloader. The machdep.svs.enabled sysctl becomes read-only, and just
indicates whether SVS is enabled.

Sent on port-amd64@.

Rename the PTE bits.

In fact, xc_broadcast also applies to offline CPUs, so we don't need to
make sure each CPU is online. Remove the checks, I suspect they weren't
totally correct by the way.

Start replacing the x86 PTE bits.

Sync with HEAD, resolve a few conflicts

Simplify, use _pi instead of modulos, no real functional change.

Sync with HEAD, resolve a couple of conflicts

Rename 'mask' -> 'frame', we will use the real 'mask' soon.

Sync with HEAD

Resolve a couple of conflicts (result of the uimin/uimax changes)

Introduce PDIR_SLOT_USERLIM, which indicates the limit of the user slots.
Use it instead of PDIR_SLOT_PTE when we just want to iterate over the
user slots. Also use it in SVS, I had hardcoded 255 because there was no
proper define (which there now is).

Sync with HEAD

Handle NMIs correctly when SVS is enabled. We store the kernel's CR3 at the
top of the NMI stack, and we unconditionally switch to it, because we don't
know with which page tables we received the NMI. Hotpatch the whole thing as
usual.

This restores the ability to use PMCs on Intel CPUs.

Pull up following revision(s) (requested by alnsn in ticket #790):

	share/man/man9/secmodel_securelevel.9: revision 1.16
	sys/secmodel/suser/secmodel_suser.c: revision 1.44
	sys/secmodel/securelevel/secmodel_securelevel.c: revision 1.31
	sys/sys/kauth.h: revision 1.76
	sys/arch/x86/x86/svs.c: revision 1.18

Add KAUTH_MACHDEP_SVS_DISABLE and add support to secmodel_securelevel(9).
Disabling SVS is denied at securelevel 1 and above.

Add SVS. It may not be disabled at securelevel 1 and above.

Synch with HEAD

Add KAUTH_MACHDEP_SVS_DISABLE and add support to secmodel_securelevel(9).

Disabling SVS is denied at securelevel 1 and above.

Sync with HEAD.  77 conflicts resolved - all of them $NetBSD$

Pull up the following revisions, requested by maxv in ticket #683:

	sys/arch/x86/x86/svs.c          1.15-1.17

Fix sysctl type, should be bool.

Use EOPNOTSUPP instead of EINVAL.

Improve the detection. Future generations of Intel CPUs will have a bit to
say they are not affected by Meltdown.

Improve the detection. Future generations of Intel CPUs will have a bit to
say they are not affected by Meltdown.

Resolve conflicts between branch and HEAD

Use EOPNOTSUPP instead of EINVAL.

Fix sysctl type, should be bool.

Pull up the following revisions, requested by maxv in ticket #652:

	sys/arch/amd64/amd64/amd64_trap.S	upto 1.39 (partial, patch)
	sys/arch/amd64/amd64/db_machdep.c	1.6 (patch)
	sys/arch/amd64/amd64/genassym.cf	1.65,1.66,1.67 (patch)
	sys/arch/amd64/amd64/locore.S		upto 1.159 (partial, patch)
	sys/arch/amd64/amd64/machdep.c		1.299-1.302 (patch)
	sys/arch/amd64/amd64/trap.c		upto 1.113 (partial, patch)
	sys/arch/amd64/amd64/amd64/vector.S	upto 1.61 (partial, patch)
	sys/arch/amd64/conf/GENERIC		1.477,1.478 (patch)
	sys/arch/amd64/conf/kern.ldscript	1.26 (patch)
	sys/arch/amd64/include/frameasm.h	upto 1.37 (partial, patch)
	sys/arch/amd64/include/param.h		1.25 (patch)
	sys/arch/amd64/include/pmap.h		1.41,1.43,1.44 (patch)
	sys/arch/x86/conf/files.x86		1.91,1.93 (patch)
	sys/arch/x86/include/cpu.h		1.88,1.89 (patch)
	sys/arch/x86/include/pmap.h		1.75 (patch)
	sys/arch/x86/x86/cpu.c			1.144,1.146,1.148,1.149 (patch)
	sys/arch/x86/x86/pmap.c			upto 1.289 (partial, patch)
	sys/arch/x86/x86/vm_machdep.c		1.31,1.32 (patch)
	sys/arch/x86/x86/x86_machdep.c		1.104,1.106,1.108 (patch)
	sys/arch/x86/x86/svs.c			1.1-1.14
	sys/arch/xen/conf/files.compat		1.30 (patch)

Backport SVS. Not enabled yet.

Synch with HEAD

file svs.c was added on branch netbsd-8 on 2018-03-22 16:59:04 +0000

Mmh, add a missing x86_disable_intr(). My intention there was to ensure
interrupts were disabled before the barriers.

Remove these two KASSERTs. Thinking about it, they may fire when the user
enters "sysctl -w machdep.svs.enabled=0", if the xcall is received between
the 'svs_enabled' check in the caller and the same check in these KASSERTs.

In such a case we perform an SVS operation with svs_enabled set to false,
but that's intentional: after it is done svs_pmap_sync and svs_lwp_switch
won't be called anymore, the pdir synchronization is dropped.

Having said that, I didn't see these KASSERTs getting triggered.

Remove the first entry from the todo list, it's handled properly now.

Fix one thing in the documentation, I meant to say only SVS_UTLS.

Document SVS. Also, remove an entry from the todo list.

Add a new entry in the TODO list.

Remove svs_pgg_update(). Instead of manually changing PG_G on each page,
we can disable the global-paging mechanism in %cr4 with CR4_PGE. Do that.

In addition, install CR4_PGE when SVS is disabled manually (via the
sysctl).

Now, doing "sysctl -w machdep.svs_enabled=0" restores the performance
completely, exactly as if SVS hadn't been enabled in the first place.

Ensure the CPUs are all online. We take cpu_lock, so nobody can go offline
in the meantime.

Make the machdep.svs_enabled sysctl writable, and add the kernel code
needed to disable SVS at runtime.

We set 'svs_enabled' to false, and hotpatch the kernel entry/exit points
to eliminate the context switch code.

We need to make sure there is no remote CPU that is executing the code we
are hotpatching. So we use two barriers:

 * After the first one each CPU is guaranteed to be executing in
   svs_disable_cpu with interrupts disabled (this way it can't leave this
   place).

 * After the second one it is guaranteed that SVS is disabled, so we flush
   the cache, enable interrupts and continue execution normally.

Between the two barriers, cpu0 will disable SVS (svs_enabled=false and
hotpatch), and each CPU will restore the generic syscall entry point.

Three notes:

 * We should call svs_pgg_update(true) afterwards, to put back PG_G on
   the kernel pages (for better performance). This will be done in another
   commit.

 * The fact that we disable interrupts does not prevent us from receiving
   an NMI, and it would be problematic. So we need to add some code to
   verify that PMCs are disabled before hotpatching. This will be done
   in another commit.

 * In svs_disable() we expect each CPU to be online. We need to add a
   check to make sure they indeed are.

The sysctl allows only a 1->0 transition. There is no point in doing 0->1
transitions anyway, and it would be complicated to implement because we
need to re-synchronize the CPU user page tables with the current ones (we
lost track of them in the last 1->0 transition).

Improve the SVS initialization.

Declare x86_patch_window_open() and x86_patch_window_close(), and globalify
x86_hotpatch().

Introduce svs_enable() in x86/svs.c, that does the SVS hotpatching.

Change svs_init() to take a bool. This function gets called twice; early
when the system just booted (and nothing is initialized), lately when at
least pmap_kernel has been initialized.

Add a dynamic detection for SVS.

The SVS_* macros are now compiled as skip-noopt. When the system boots, if
the cpu is from Intel, they are hotpatched to their real content.
Typically:

		jmp	1f
		int3
		int3
		int3
		... int3 ...
	1:

gets hotpatched to:

		movq	SVS_UTLS+UTLS_KPDIRPA,%rax
		movq	%rax,%cr3
		movq	CPUVAR(KRSP0),%rsp

These two chunks of code being of the exact same size. We put int3 (0xCC)
to make sure we never execute there.

In the non-SVS (ie non-Intel) case, all it costs is one jump. Given that
the SVS_* macros are small, this jump will likely leave us in the same
icache line, so it's pretty fast.

The syscall entry point is special, because there we use a scratch uint64_t
not in curcpu but in the UTLS page, and it's difficult to hotpatch this
properly. So instead of hotpatching we declare the entry point as an ASM
macro, and define two functions: syscall and syscall_svs, the latter being
the one used in the SVS case.

While here 'syscall' is optimized not to contain an SVS_ENTER - this way
we don't even need to do a jump on the non-SVS case.

When adding pages in the user page tables, make sure we don't have PG_G,
now that it's dynamic.

A read-only sysctl is added, machdep.svs_enabled, that tells whether the
kernel uses SVS or not.

More changes to come, svs_init() is not very clean.

Add svs_enabled, which defaults to 'true' when SVS is compiled (no dynamic
detection yet).

Add svs_init. This is where we will detect the CPU and decide whether
to turn SVS on or not.

Add svs_pgg_update to dynamically add/remove PG_G from all the kernel
pages. Use it now.

Move SVS into x86/svs.c