CVS log for src/sys/arch/amd64/amd64/amd64_trap.S

Up to [cvs.NetBSD.org] / src / sys / arch / amd64 / amd64

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

Pull up following revision(s) (requested by rin in ticket #876):

	sys/arch/amd64/amd64/amd64_trap.S: revision 1.55

amd64_trap.S: Annotate trap vectors with their mnemonics.

Derived from Intel and AMD manuals.
Intel 64 and IA-32 Architectures Software Developer's Manual, Volume 3
(3A, 3B, 3C, & 3D: System Programming Guide, Order Number:
325384-077US, April 2022.
https://cdrdv2.intel.com/v1/dl/getContent/671447

AMD64 Technology: AMD64 Architecture Programmers' Manual, Volume 2:
System Programming, Publication No. 24953, Revision 3.40, January 2023.
https://www.amd.com/system/files/TechDocs/24593.pdf

No functional change intended.

amd64_trap.S: Annotate trap vectors with their mnemonics.

Derived from Intel and AMD manuals.

Intel 64 and IA-32 Architectures Software Developer's Manual, Volume 3
(3A, 3B, 3C, & 3D: System Programming Guide, Order Number:
325384-077US, April 2022.
https://cdrdv2.intel.com/v1/dl/getContent/671447

AMD64 Technology: AMD64 Architecture Programmers' Manual, Volume 2:
System Programming, Publication No. 24953, Revision 3.40, January 2023.
https://www.amd.com/system/files/TechDocs/24593.pdf

No functional change intended.

XXX Should apply the same treatment to i386_trap.S.

NetBSD/x86: Raise the number of interrupt sources per CPU from 32 to 56.

There has been no objection for three years.
    https://mail-index.netbsd.org/port-amd64/2019/09/22/msg003012.html
Implemented by nonaka@n.o, updated by me.

Nix trailing whitespace.

On amd64, Xen PV calls syscalls and traps with events enabled.
Disable events on entry to be safe.
It should have been mostly safe for most cases, but for FPU traps
we need to reload the FPU state if we got interrupted at trap entry.

Hopefully fixes:
panic: kernel diagnostic assertion "curlwp->l_md.md_flags & MDL_FPU_IN_CPU" failed: file "/home/source/ab/HEAD/src/sys/arch/x86/x86/fpu.c", line 524

when running tests.

Mostly merge changes from HEAD upto 20200411

Merge changes from current as of 20200406

Panic instead of printf, same as syscall.

Add support for Kernel Memory Sanitizer (kMSan). It detects uninitialized
memory used by the kernel at run time, and just like kASan and kCSan, it
is an excellent feature. It has already detected 38 uninitialized variables
in the kernel during my testing, which I have since discreetly fixed.

We use two shadows:
 - "shad", to track uninitialized memory with a bit granularity (1:1).
   Each bit set to 1 in the shad corresponds to one uninitialized bit of
   real kernel memory.
 - "orig", to track the origin of the memory with a 4-byte granularity
   (1:1). Each uint32_t cell in the orig indicates the origin of the
   associated uint32_t of real kernel memory.

The memory consumption of these shadows is consequent, so at least 4GB of
RAM is recommended to run kMSan.

The compiler inserts calls to specific __msan_* functions on each memory
access, to manage both the shad and the orig and detect uninitialized
memory accesses that change the execution flow (like an "if" on an
uninitialized variable).

We mark as uninit several types of memory buffers (stack, pools, kmem,
malloc, uvm_km), and check each buffer passed to copyout, copyoutstr,
bwrite, if_transmit_lock and DMA operations, to detect uninitialized memory
that leaves the system. This allows us to detect kernel info leaks in a way
that is more efficient and also more user-friendly than KLEAK.

Contrary to kASan, kMSan requires comprehensive coverage, ie we cannot
tolerate having one non-instrumented function, because this could cause
false positives. kMSan cannot instrument ASM functions, so I converted
most of them to __asm__ inlines, which kMSan is able to instrument. Those
that remain receive special treatment.

Contrary to kASan again, kMSan uses a TLS, so we must context-switch this
TLS during interrupts. We use different contexts depending on the interrupt
level.

The orig tracks precisely the origin of a buffer. We use a special encoding
for the orig values, and pack together in each uint32_t cell of the orig:
 - a code designating the type of memory (Stack, Pool, etc), and
 - a compressed pointer, which points either (1) to a string containing
   the name of the variable associated with the cell, or (2) to an area
   in the kernel .text section which we resolve to a symbol name + offset.

This encoding allows us not to consume extra memory for associating
information with each cell, and produces a precise output, that can tell
for example the name of an uninitialized variable on the stack, the
function in which it was pushed on the stack, and the function where we
accessed this uninitialized variable.

kMSan is available with LLVM, but not with GCC.

The code is organized in a way that is similar to kASan and kCSan, so it
means that other architectures than amd64 can be supported.

Rewrite the FPU code on x86. This greatly simplifies the logic and removes
the dependency on IPL_HIGH. NVMM is updated accordingly. Posted on
port-amd64 a week ago.

Bump the kernel version to 9.99.16.

Sync with HEAD

Two changes in the CPU mitigations:

 * Micro-optimize: put every mitigation in the same branch. This removes
   two branches in each exc/int return path, and removes all branches in
   the syscall return path.

 * Modify the SpectreV2 mitigation to be compatible with SpectreV4. I
   recently realized that both couldn't be enabled at the same time on
   Intel. This is because initially, when there was just SpectreV2, we
   could reset the whole IA32_SPEC_CTRL MSR. But then Intel added another
   bit in it for SpectreV4, so it isn't right to reset it entirely
   anymore. SSBD needs to stay.

Pull up following revision(s) (requested by maxv in ticket #1269):

	sys/arch/amd64/amd64/locore.S: revision 1.181 (adapted)
	sys/arch/amd64/amd64/amd64_trap.S: revision 1.47 (adapted)
	sys/arch/x86/include/specialreg.h: revision 1.144 (adapted)
	sys/arch/amd64/include/frameasm.h: revision 1.43 (adapted)
	sys/arch/x86/x86/spectre.c: revision 1.27 (adapted)

Mitigation for INTEL-SA-00233: Microarchitectural Data Sampling (MDS).
It requires a microcode update, now available on the Intel website. The
microcode modifies the behavior of the VERW instruction, and makes it flush
internal CPU buffers. We hotpatch the return-to-userland path to add VERW.

Two sysctls are added:

	machdep.mds.mitigated = {0/1} user-settable
	machdep.mds.method = {string} constructed by the kernel

The kernel will automatically enable the mitigation if the updated
microcode is present. If the new microcode is not present, the user can
load it via cpuctl, and set machdep.mds.mitigated=1.

Mitigation for INTEL-SA-00233: Microarchitectural Data Sampling (MDS).

It requires a microcode update, now available on the Intel website. The
microcode modifies the behavior of the VERW instruction, and makes it flush
internal CPU buffers. We hotpatch the return-to-userland path to add VERW.

Two sysctls are added:

	machdep.mds.mitigated = {0/1} user-settable
	machdep.mds.method = {string} constructed by the kernel

The kernel will automatically enable the mitigation if the updated
microcode is present. If the new microcode is not present, the user can
load it via cpuctl, and set machdep.mds.mitigated=1.

We reorganise definitions for XEN source support as follows:

XEN - common sources required for baseline XEN support.
XENPV - sources required for support of XEN in PV mode.
XENPVHVM - sources required for support for XEN in HVM mode.
XENPVH - sources required for support for XEN in PVH mode.

Sync with HEAD

Resolve a couple of conflicts (result of the uimin/uimax changes)

Eliminate the only ASM reference to VM_MIN_KERNEL_ADDRESS. Rename the
value to VM_SPACE_SEP_HIGH32, it is now the highest 32bits of the first
va of the higher half of the address space (right after the canonical
hole).

Sync with HEAD

Drop NENTRY() from the x86 kernels, use ENTRY(). With PMCs (and other hardware
tracing facilities) we have a much better ways of monitoring the CPU activity
than GPROF, without software modification.

Also I think GPROF has never worked, because the 'start' functions of both
i386 and amd64 use ENTRY(), and it would have caused a function call while the
kernel was not yet relocated.

Handle NMIs correctly when SVS is enabled. We store the kernel's CR3 at the
top of the NMI stack, and we unconditionally switch to it, because we don't
know with which page tables we received the NMI. Hotpatch the whole thing as
usual.

This restores the ability to use PMCs on Intel CPUs.

Sync with HEAD

When the previous contrext is in kernel mode we are not guaranteed to have
a 16-byte-aligned stack pointer, so align it. That's what the CPU would do
on exception entry.

Sync with HEAD

Mitigation for the SS bug, CVE-2018-8897. We disabled dbregs a month ago
in -current and -8 so we are not particularly affected anymore.

The #DB handler runs on ist3, if we decide to process the exception we
copy the iret frame on the correct non-ist stack and continue as usual.

Pullup the following revisions via patch, requested by maxv in ticket #748:

sys/arch/amd64/amd64/copy.S		1.29 (adapted, via patch)
sys/arch/amd64/amd64/amd64_trap.S	1.16,1.19 (partial) (via patch)
sys/arch/amd64/amd64/trap.c		1.102,1.106 (partial),1.110 (via patch)
sys/arch/amd64/include/frameasm.h	1.22,1.24 (via patch)
sys/arch/x86/x86/cpu.c			1.137 (via patch)
sys/arch/x86/x86/patch.c		1.23,1.26 (partial) (via patch)

Backport of SMAP support.

Resolve conflicts between branch and HEAD

Add the IBRS mitigation for SpectreV2 on amd64.

Different operations are performed during context transitions:

	user->kernel: IBRS <- 1
	kernel->user: IBRS <- 0

And during context switches:

	user->user:   IBPB <- 0
	kernel->user: IBPB <- 0
	[user->kernel:IBPB <- 0 this one may not be needed]

We use two macros, IBRS_ENTER and IBRS_LEAVE, to set the IBRS bit. The
thing is hotpatched for better performance, like SVS.

The idea is that IBRS is a "privileged" bit, which is set to 1 in kernel
mode and 0 in user mode. To protect the branch predictor between user
processes (which are of the same privilege), we use the IBPB barrier.

The Intel manual also talks about (MWAIT/HLT)+HyperThreading, and says
that when using either of the two instructions IBRS must be disabled for
better performance on the core. I'm not totally sure about this part, so
I'm not adding it now.

IBRS is available only when the Intel microcode update is applied. The
mitigation must be enabled manually with machdep.spectreV2.mitigated.

Tested by msaitoh a week ago (but I adapted a few things since). Probably
more changes to come.

Pull up the following revisions, requested by maxv in ticket #652:

	sys/arch/amd64/amd64/amd64_trap.S	upto 1.39 (partial, patch)
	sys/arch/amd64/amd64/db_machdep.c	1.6 (patch)
	sys/arch/amd64/amd64/genassym.cf	1.65,1.66,1.67 (patch)
	sys/arch/amd64/amd64/locore.S		upto 1.159 (partial, patch)
	sys/arch/amd64/amd64/machdep.c		1.299-1.302 (patch)
	sys/arch/amd64/amd64/trap.c		upto 1.113 (partial, patch)
	sys/arch/amd64/amd64/amd64/vector.S	upto 1.61 (partial, patch)
	sys/arch/amd64/conf/GENERIC		1.477,1.478 (patch)
	sys/arch/amd64/conf/kern.ldscript	1.26 (patch)
	sys/arch/amd64/include/frameasm.h	upto 1.37 (partial, patch)
	sys/arch/amd64/include/param.h		1.25 (patch)
	sys/arch/amd64/include/pmap.h		1.41,1.43,1.44 (patch)
	sys/arch/x86/conf/files.x86		1.91,1.93 (patch)
	sys/arch/x86/include/cpu.h		1.88,1.89 (patch)
	sys/arch/x86/include/pmap.h		1.75 (patch)
	sys/arch/x86/x86/cpu.c			1.144,1.146,1.148,1.149 (patch)
	sys/arch/x86/x86/pmap.c			upto 1.289 (partial, patch)
	sys/arch/x86/x86/vm_machdep.c		1.31,1.32 (patch)
	sys/arch/x86/x86/x86_machdep.c		1.104,1.106,1.108 (patch)
	sys/arch/x86/x86/svs.c			1.1-1.14
	sys/arch/xen/conf/files.compat		1.30 (patch)

Backport SVS. Not enabled yet.

Synch with HEAD, resolve conflicts

(Re)Fix handling of segment register faults. My previous attempt did fix
faults occuring when reloading %es/%ds/%fs/%gs, but it did not fix faults
occuring when executing 'iretq', because before iretq we needed to do +16
in %rsp, and the resulting stack layout was not the one kernuser_reenter()
expected (tf_trapno and tf_err were not there).

So now: pop tf_trapno and tf_err right away in intrfastexit(), and update
the layout in kernuser_reenter() accordingly. The resulting code is
actually simpler.

Tested by "hardcoding" an iretq fault; the process correctly receives a
SIGSEGV.

(Note that segment register faults do not happen in the wild, you really
need to try hard to trigger one.)

Remove the sysretq fault handler. It is broken with SVS, and not really
needed anyway. Initially I had added it so that if such a fault was
received the kernel would panic "cleanly" instead of crashing in a
potentially undefined way.

I'll re-add this handler later.

Pull up the following revisions (via patch), requested by maxv in ticket #610:

sys/arch/amd64/amd64/amd64_trap.S	1.8,1.10,1.12 (partial),1.13-1.15,
					1.19 (partial),1.20,1.21,1.22,1.24
					(via patch)
sys/arch/amd64/amd64/locore.S		1.129 (partial),1.132 (via patch)
sys/arch/amd64/amd64/trap.c		1.97 (partial),1.111 (via patch)
sys/arch/amd64/amd64/vector.S		1.54,1.55 (via patch)
sys/arch/amd64/include/frameasm.h	1.21,1.23 (via patch)
sys/arch/x86/x86/cpu.c			1.138 (via patch)
sys/arch/xen/conf/Makefile.xen		1.45 (via patch)

Rename and reorder several things in amd64_trap.S.
Compile amd64_trap.S as a file.
Introduce nmitrap and doubletrap.
Have the CPU clear PSL_D automatically in the syscall entry point.

Fix handling of segment register faults when running with SVS. The behavior
is changed also in the non-SVS case.

I've put a documentation in amd64_trap.S. Basically, the problem with SVS
is that if iret faults, we already have a full trapframe pushed on the
stack and the CPU will push another frame on this stack (nested), but it
hits the redzone below the stack since it is still running with the user
page table loaded.

To fix that, we pop a good part of the trapframe earlier in intrfastexit.
If iret faults, the current %rsp has enough room for an iret frame, and
the CPU can push that without problem. We then switch back to the outer
iret frame (the frame the CPU was trying to pop by executing iret, but that
it didn't pop for real because iret faulted), call INTRENTRY, and handle
the trap as if it had been received from userland directly.

Ah. Don't use NENTRY() to declare check_swapgs, use LABEL() instead. NENTRY
puts the code in the .text section, so the effect of TEXT_USER_BEGIN was
overwritten, and check_swapgs was not put in the .text.user section.

As a result kernels running SVS would crash when jumping here - because we
execute this place with the user page table loaded, and in this page table
only .text.user is mapped.

While here, rename check_swapgs -> kernuser_reenter, because we do more
things than just SWAPGS.

Replace %rax -> %rdi, so that check_swapgs clobbers only one register.

There are two places where we reload %gs:

 * In setusergs. Here we can't fault. So we don't need to handle this
   case.

 * In intrfastexit for 32bit processes. This case needs to be handled,
   and we already have a label.

So use the label instead of disassembling %rip.

Revert all my latest changes, and restore this file back to how it was
in rev1.24. I wanted to replace the functions dynamically for SVS, but
that was a dumb idea, we'll just hotpatch instead.

Pass the name of the function as argument in SWAPGS_HANDLER.

Declare check_swapgs in an ASM macro. No real functional change.

Use ASM macros for the rest of the entry points. No real functional
change. Now the format of the entry points is:

	.macro	TRAP_ENTRY_POINT_xx	arg1,arg2,arg3
		...the asm code...
	.endm

		TEXT_USER_BEGIN
	TRAP_ENTRY_POINT_xx	arg1,arg2,arg3
		TEXT_USER_END

Declare and use TRAP_ENTRY_POINT_DNA. This time we don't give an is_ztrap
argument, because the macro is tied to the entry point, and it would be
wrong to suggest the paramater is controllable. No real functional change.

Now that [Z]TRAP and [Z]TRAP_NJ are identical, put back the

	INTRENTRY
	jmp	.Lalltraps_noentry

instructions for Xen, and remove [Z]TRAP_NJ.

Declare and use TRAP_ENTRY_POINT_SPUR. No real functional change.

Declare and use TRAP_ENTRY_POINT_FPU. No real functional change.

Start using ASM macros to define the trap entry points. No real functional
change.

Don't restore segment registers when leaving NMIs. In nmitrap (and the
functions it later calls), we are not allowing the trap frame to change;
so the segregs don't change since we are running with interrupts disabled
and there is no rescheduling in this case.

Unmap the kernel from userland in SVS, and leave only the needed
trampolines. As explained below, SVS should now completely mitigate
Meltdown on GENERIC kernels, even though it needs some more tweaking
for GENERIC_KASLR.

Until now the kernel entry points looked like:

	FUNC(intr)
		pushq	$ERR
		pushq	$TRAPNO
		INTRENTRY
		... handle interrupt ...
		INTRFASTEXIT
	END(intr)

With this change they are split and become:

	FUNC(handle)
		... handle interrupt ...
		INTRFASTEXIT
	END(handle)

		TEXT_USER_BEGIN
	FUNC(intr)
		pushq	$ERR
		pushq	$TRAPNO
		INTRENTRY
		jmp	handle
	END(intr)
		TEXT_USER_END

A new section is introduced, .text.user, that contains minimal kernel
entry/exit points. In order to choose what to put in this section, two
macros are introduced, TEXT_USER_BEGIN and TEXT_USER_END.

The section is mapped in userland with normal 4K pages.

In GENERIC, the section is 4K-page-aligned and embedded in .text, which
is mapped with large pages. That is to say, when an interrupt comes in,
the CPU has the user page tables loaded and executes the 'intr' functions
on 4K pages; after calling SVS_ENTER (in INTRENTRY) these 4K pages become
2MB large pages, and remain so when executing in kernel mode.

In GENERIC_KASLR, the section is 4K-page-aligned and independent from the
other kernel texts. The prekern just picks it up and maps it at a random
address.

In GENERIC, SVS should now completely mitigate Meltdown: what we put in
.text.user is not secret.

In GENERIC_KASLR, SVS would have to be improved a bit more: the
'jmp handle' instruction is actually secret, since it leaks the address
of the section we are jumping into. By exploiting Meltdown on Intel, this
theoretically allows a local user to reconstruct the address of the first
text section. But given that our KASLR produces several texts, and that
each section is not correlated with the others, the level of protection
KASLR provides is still good.

Compile amd64_trap.S as a file instead of including it.

Eliminate a '.text'.

Don't declare exceptions[] with IDTVEC, it's an array, not a function.
Rename it to x86_exceptions[], and move it to .rodata.

Fix the double-fault handler. We're executing on ist1 and must not jump
out of it, so don't enable interrupts. And use the SVS_*_ALTSTACK macros.

While here, fix the NMI handler too: it should use SVS_LEAVE_ALTSTACK.

Unmap the kernel heap from the user page tables (SVS).

This implementation is optimized and organized in such a way that we
don't need to copy the kernel stack to a safe place during user<->kernel
transitions. We create two VAs that point to the same physical page; one
will be mapped in userland and is offset in order to contain only the
trapframe, the other is mapped in the kernel and maps the entire stack.

Sent on tech-kern@ a week ago.

Add a new option, SVS (for Separate Virtual Space), that unmaps kernel
pages when running in userland. For now, only the PTE area is unmapped.

Sent on tech-kern@.

Implement a real hotpatch feature.

Define a HOTPATCH() macro, that puts a label and additional information
in the new .rodata.hotpatch kernel section. In patch.c, scan the section
and patch what needs to be. Now it is possible to hotpatch the content of
a macro.

SMAP is switched to use this new system; this saves a call+ret in each
kernel entry/exit point.

Many other operating systems do the same.

Mmh, I made a mistake in r1.10 - I forgot to update this function call.

Use decimal numbering - hex is just misleading -, use ZTRAP_NJ for NMIs,
and declare intrspurious independently.

update from HEAD

Hide a bunch of raw symbols.

Have the cpu clear PSL_D automatically when entering the kernel via a
syscall. Then, don't clear PSL_D and PSL_AC in the syscall entry point,
they are now both cleared by the cpu (faster). However they still need to
be manually cleared in the interrupt/trap entry points.

Declare INTRFASTEXIT as a function, so that there is only one iretq in the
kernel. Then, check %rip against the address of this iretq instead of
disassembling (%rip) - which could fault again, or point at some random
address which happens to contain the iretq opcode. The same is true for gs
below, but I'll fix that in another commit.

Remove useless debug code, and split trap() into smaller functions, easier
to understand. NMIs take another, faster path now. No functional change
beyond that.

Add a layer of mitigation against the intel sysret vuln: restore %gs when
sysretq faults. Right now we try to make sure that %rip is canonical by
performing sanity checks in several places, but I've already found missing
checks two times already, and there may be others.

By performing an additional swapgs here, we are turning ring0 exploits to
simple DoSes - which are still security bugs, but of a lower impact.

Reorder for clarity, and style.

Sync with HEAD

Revert my previous change. I hadn't checked carefully enough: the
symbols are used in src/external. There is a number of things that seem
wrong to me here, but I'm not changing them for now.

Remove unused and broken code. On amd64 we won't want int3 from kernel
mode to be valid.

Sync with HEAD

Sync with HEAD

Unconditionnally save the segment registers - because we could have a
kernel %gs and a userland %es/%ds -, and explain why T_NMI is a special
case.

Note that checking %gs directly is not a good idea: recent CPUs have the
FSGSBASE instruction set, which allows userland to directly modify %gs
without going through the kernel. If we ever enable this set, we will have
to change this function, since we won't be able to test %gs against
VM_MIN_KERNEL_ADDRESS anymore.

Sync with HEAD

Explain a little.

Sync with HEAD (as of 26th Dec)

KNF a bit, so I don't get scared each time I open a file

Rebase to HEAD as of a few days ago.

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs.  ("Protocol error: too many arguments")

sync with head

file amd64_trap.S was added on branch tls-maxphys on 2014-08-20 00:02:42 +0000

file amd64_trap.S was added on branch yamt-pagecache on 2014-05-22 11:39:28 +0000

Change the argument to fpudna() to be the trapframe.
Move the checks for fpu traps in kernel into x86/fpu.c.
Remove the code from amd64/trap.c related to fpu traps (they've not gone
  there for ages - expect to panic in kernel mode).
In fpudna():
- Don't actually enable hardware interrupts unless we need to
  allow in IPIs.
- There is no point in enabling them when they are blocked in software
  (by splhigh()).
- Keep the splhigh() to avoid a load of the KASSERTS() firing.

Split these to improve diffability.