CVS log for src/sys/arch/x86/x86/patch.c

Up to [cvs.NetBSD.org] / src / sys / arch / x86 / x86

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

x86: Split most of pmap.h into pmap_private.h or vmparam.h.

This way pmap.h only contains the MD definition of the MI pmap(9)
API, which loads of things in the kernel rely on, so changing x86
pmap internals no longer requires recompiling the entire kernel every
time.

Callers needing these internals must now use machine/pmap_private.h.
Note: This is not x86/pmap_private.h because it contains three parts:

1. CPU-specific (different for i386/amd64) definitions used by...

2. common definitions, including Xenisms like xpmap_ptetomach,
   further used by...

3. more CPU-specific inlines for pmap_pte_* operations

So {amd64,i386}/pmap_private.h defines 1, includes x86/pmap_private.h
for 2, and then defines 3.  Maybe we should split that out into a new
pmap_pte.h to reduce this trouble.

No functional change intended, other than that some .c files must
include machine/pmap_private.h when previously uvm/uvm_pmap.h
polluted the namespace with pmap internals.

Note: This migrates part of i386/pmap.h into i386/vmparam.h --
specifically the parts that are needed for several constants defined
in vmparam.h:

VM_MAXUSER_ADDRESS
VM_MAX_ADDRESS
VM_MAX_KERNEL_ADDRESS
VM_MIN_KERNEL_ADDRESS

Since i386 needs PDP_SIZE in vmparam.h, I added it there on amd64
too, just to keep things parallel.

x86: Split bootspace out of x86/pmap.h into new x86/bootspace.h.

x86: Eliminate mfence hotpatch for membar_sync.

The more-compatible  LOCK ADD $0,-N(%rsp)  turns out to be cheaper
than MFENCE anyway.  Let's save some space and maintenance and rip
out the hotpatching for it.

x86: Every load is a load-acquire, so membar_consumer is a noop.

lfence is only needed for MD logic, such as operations on I/O memory
rather than normal cacheable memory, or special instructions like
RDTSC -- never for MI synchronization between threads/CPUs.  No need
for hot-patching to do lfence here.

(The x86_lfence function might reasonably be patched on i386 to do
lfence for MD logic, but it isn't now and this doesn't change that.)

Fix LOCKDEBUG compilation on i386.

Remove the D bit as part of the hotpatch cleanup procedure.

Modify the hotpatch mechanism, in order to make it much less ROP-friendly.

Currently x86_patch_window_open is a big problem, because it is a perfect
function to inject/modify executable code with ROP.

 - Remove x86_patch_window_open(), along with its x86_patch_window_close()
   counterpart.
 - Introduce a read-only link-set of hotpatch descriptor structures,
   which reference a maximum of two read-only hotpatch sources.
 - Modify x86_hotpatch() to open a window and call the new
   x86_hotpatch_apply() function in a hard-coded manner.
 - Modify x86_hotpatch() to take a name and a selector, and have
   x86_hotpatch_apply() resolve the descriptor from the name and the
   source from the selector, before hotpatching.
 - Move the error handling in a separate x86_hotpatch_cleanup() function,
   that gets called after we closed the window.

The resulting implementation is a bit complex and non-obvious. But it
gains the following properties: the code executed in the hotpatch window
is strictly hard-coded (no callback and no possibility to execute your own
code in the window) and the pointers this code accesses are strictly
read-only (no possibility to forge pointers to hotpatch an area that was
not designated as hotpatchable at compile-time, and no possibility to
choose what bytes to write other than the maximum of two read-only
templates that were designated as valid for the given destination at
compile-time).

With current CPUs this slightly improves a situation that is already
pretty bad by definition on x86. Assuming CET however, this change closes
a big hole and is kinda great.

The only ~problem there is, is that dtrace-fbt tries to hotpatch random
places with random bytes, and there is just no way to make it safe.
However dtrace is only in a module, that is rarely used and never compiled
into the kernel, so it's not a big problem; add a shitty & vulnerable
independent hotpatch window in it, and leave big XXXs. It looks like fbt
is going to collapse soon anyway.

Switch the rest of i386 to the x86_hotpatch mechanism.

Use absolute jumps, and drop the PC-relative patching. We want exact
templates.

Use the hotpatch framework when patching _atomic_cas_64.

Switch to templates.

Use the hotpatch framework for LFENCE/MFENCE.

Drop the hardcoded array, use the hotpatch section.

Merge the bouyer-xenpvh branch, bringing in Xen PV drivers support under HVM
guests in GENERIC.
Xen support can be disabled at runtime with
boot -c
disable hypervisor

On amd64, always use the cmpxchg8b version of spllower. All x86_64 host should
have it and we already rely on it in lock stubs.
On i386, always use i686_mutex_spin_exit and cx8_spllower for Xen;
Xen doesn't run on CPUs on CPUs lacking the required instructions anyway.
Skip x86_patch only for XENPV, and adjust for changes in assembly functions.
Tested on Xen PV and PVHVM, and on bare metal core i5.

Always patch spllower with cx8_spllower; it works fine for Xen now
Include x86/x86/patch.c is !xenpv
While there, defopt XENPV

Mostly merge changes from HEAD upto 20200411

Skip cx8_spllower patch if we're running on any form of Xen PV,
we can't handle PV interrupts with a single atomic op here.
Enable x86_patch() for Xen too.

Revert, wrong branch

Skip cx8_spllower patch if we're running on any form of Xen PV,
we can't handle PV interrupts with a single atomic op here.
Enable x86_patch() for Xen too.

Switch to __noubsan in x86_hotpatch()

Sync with HEAD

Disable sanitizer instrumentation in x86_hotpatch()

Local variables have empty (0-sized), unknown alignment to UBSan.
This is hard to workaround without mutating the code too much.

Sync with HEAD

Remove ifdef GPROF.

Pullup the following revisions via patch, requested by maxv in ticket #748:

sys/arch/amd64/amd64/copy.S		1.29 (adapted, via patch)
sys/arch/amd64/amd64/amd64_trap.S	1.16,1.19 (partial) (via patch)
sys/arch/amd64/amd64/trap.c		1.102,1.106 (partial),1.110 (via patch)
sys/arch/amd64/include/frameasm.h	1.22,1.24 (via patch)
sys/arch/x86/x86/cpu.c			1.137 (via patch)
sys/arch/x86/x86/patch.c		1.23,1.26 (partial) (via patch)

Backport of SMAP support.

Synch with HEAD

Fix wrong order; first enable WP, then enable interrupts. Otherwise we
might get an interrupt before re-enabling WP, and be rescheduled as a
result. In practice it never happens, because the previous PSL always
has interrupts disabled too.

Pull up the following revisions, requested by maxv in ticket #603:

	amd64/conf/kern.ldscript      1.25 (patch)
	amd64/conf/kern.ldscript.Xen  1.14 (patch)
	i386/conf/kern.ldscript       1.21 (patch)
	i386/conf/kern.ldscript.Xen   1.15 (patch)
	x86/include/cpufunc.h         1.24 (patch)
	x86/x86/patch.c               1.25 (partial) 1.26 (partial)

Backport x86_hotpatch.

Improve the SVS initialization.

Declare x86_patch_window_open() and x86_patch_window_close(), and globalify
x86_hotpatch().

Introduce svs_enable() in x86/svs.c, that does the SVS hotpatching.

Change svs_init() to take a bool. This function gets called twice; early
when the system just booted (and nothing is initialized), lately when at
least pmap_kernel has been initialized.

Add a dynamic detection for SVS.

The SVS_* macros are now compiled as skip-noopt. When the system boots, if
the cpu is from Intel, they are hotpatched to their real content.
Typically:

		jmp	1f
		int3
		int3
		int3
		... int3 ...
	1:

gets hotpatched to:

		movq	SVS_UTLS+UTLS_KPDIRPA,%rax
		movq	%rax,%cr3
		movq	CPUVAR(KRSP0),%rsp

These two chunks of code being of the exact same size. We put int3 (0xCC)
to make sure we never execute there.

In the non-SVS (ie non-Intel) case, all it costs is one jump. Given that
the SVS_* macros are small, this jump will likely leave us in the same
icache line, so it's pretty fast.

The syscall entry point is special, because there we use a scratch uint64_t
not in curcpu but in the UTLS page, and it's difficult to hotpatch this
properly. So instead of hotpatching we declare the entry point as an ASM
macro, and define two functions: syscall and syscall_svs, the latter being
the one used in the SVS case.

While here 'syscall' is optimized not to contain an SVS_ENTER - this way
we don't even need to do a jump on the non-SVS case.

When adding pages in the user page tables, make sure we don't have PG_G,
now that it's dynamic.

A read-only sysctl is added, machdep.svs_enabled, that tells whether the
kernel uses SVS or not.

More changes to come, svs_init() is not very clean.

Add SMAP support for i386.

make this compile w/o LOCKDEBUG

Switch x86_retpatch[] -> HOTPATCH().

Fix previous - atomic_lockpatch[] is still there.

Switch x86_lockpatch[] -> HOTPATCH().

Implement a real hotpatch feature.

Define a HOTPATCH() macro, that puts a label and additional information
in the new .rodata.hotpatch kernel section. In patch.c, scan the section
and patch what needs to be. Now it is possible to hotpatch the content of
a macro.

SMAP is switched to use this new system; this saves a call+ret in each
kernel entry/exit point.

Many other operating systems do the same.

Give patchbytes an array.

update from HEAD

Add comment explaining why membar_producer is not sfence.

On x86, ordinary non-temporal stores are always issued in program
order to main memory and to other CPUs.

Add support for SMAP on amd64.

PSL_AC is cleared from %rflags in each kernel entry point. In the copy
sections, a copy window is opened and the kernel can touch userland
pages. This window is closed when the kernel is done, either at the end
of the copy sections or in the fault-recover functions.

This implementation is not optimized yet, due to the fact that INTRENTRY
is a macro, and we can't hotpatch macros.

Sent on tech-kern@ a month or two ago, tested on a Kabylake.

Rebase to HEAD as of a few days ago.

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs.  ("Protocol error: too many arguments")

sync with head

 Modify some macros and add some new macros for CPU family and model
to reduce code duplication and to avoid bug.

CPUID_TO_STEPPING(cpuid)	(not changed)

CPUID_TO_FAMILY(cpuid)		(new)
CPUID_TO_MODEL(cpuid)		(new)

	Return the display family and the display model.
	The macro names are the same as FreeBSD.

CPUID_TO_BASEFAMILY(cpuid)	(The old name was CPUID2FAMILY)
CPUID_TO_BASEMODEL(cpuid)	(The old name was CPUID2MODEL)

	Only for the base field.

CPUID_TO_EXTFAMILY(cpuid)	(The old name was CPUID2EXTFAMILY)
CPUID_TO_EXTMODEL(cpuid)	(The old name was CPUID2EXTMODEL)

	Only for the extended field.

See http://mail-index.netbsd.org/port-amd64/2013/11/12/msg001978.html

Sync with HEAD

sync with head.

sync with head

Sync with HEAD.

sync to netbsd-5

This patch fixes the NX regression issue observed on amd64 kernels, where
per-page execution right was disabled (therefore leading to the inability
of the kernel to detect fraudulent use of memory mappings marked as not
being executable).

- replace cpu_feature and ci_feature_flags variables by cpu_feature and
ci_feat_val arrays. This makes it cleaner and brings kernel code closer
to the design of cpuctl(8). A warning will be raised for each CPU that
does not expose the same features as the Boot Processor (BP).

- the blacklist of CPU features is now a macro defined in the
specialreg.h header, instead of hardcoding it inside MD initialization
code; fix comments.

- replace checks against CPUID_TSC with the cpu_hascounter() function.

- clean up the code in init_x86_64(), as cpu_feature variables are set
inside cpu_probe().

- use cpu_init_msrs() for i386. It will be eventually used later for NX
feature under i386 PAE kernels.

- remove code that checks for CPUID_NOX in amd64 mptramp.S, this is already
performed by cpu_hatch() through cpu_init_msrs().

- remove cpu_signature and feature_flags members from struct mpbios_proc
(they were never used).

This patch was tested with i386 MONOLITHIC, XEN3PAE_DOM0 and XEN3_DOM0 under
a native i386 host, and amd64 GENERIC, XEN3_DOM0 via QEMU virtual machines.

XXX Should kernel rev be bumped?

XXX A similar patch should be pulled-up for NetBSD-5, hopefully tomorrow.

sync with head

Gracelessly bracket #include "opt_spldebug.h" with #ifdef i386.
Should fix the amd64 kernel-build failure that Andreas Wrede
reported.

Add a kernel configuration flag, SPLDEBUG, that activates a per-CPU log
of transitions to IPL_HIGH from lower IPLs.  SPLDEBUG is only available
on i386 and Xen kernels, today.

'options SPLDEBUG' adds instrumentation to spllower() and splraise() as
well as routines to start/stop debugging and to record IPL transitions:
spldebug_start(), spldebug_stop(), spldebug_raise(), spldebug_lower().

Sync with HEAD.

Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.

Pull up following revision(s) (requested by ad in ticket #725):
	sys/arch/amd64/amd64/lock_stubs.S: revision 1.22
	sys/arch/i386/i386/lock_stubs.S: revision 1.23
	sys/arch/x86/x86/patch.c: revision 1.18
A workaround for a bug with some Opteron revisions where locked operations
sometimes do not serve as memory barriers, allowing memory references to
bleed outside of critical sections.  It is possible that this is the
reason for pkgbuild's longstanding crashiness.
This is not complete (atomic ops need some work too).

Pull up following revision(s) (requested by ad in ticket #725):
	sys/arch/amd64/amd64/lock_stubs.S: revision 1.22
	sys/arch/i386/i386/lock_stubs.S: revision 1.23
	sys/arch/x86/x86/patch.c: revision 1.18
A workaround for a bug with some Opteron revisions where locked operations
sometimes do not serve as memory barriers, allowing memory references to
bleed outside of critical sections.  It's possible that this is the
reason for pkgbuild's longstanding crashiness.
This is not complete (atomic ops need some work too).

sync with head.

Sync with HEAD.

A workaround for a bug with some Opteron revisions where locked operations
sometimes do not serve as memory barriers, allowing memory references to
bleed outside of critical sections.  It's possible that this is the
reason for pkgbuild's longstanding crashiness.

This is not complete (atomic ops need some work too).

Pull up following revision(s) (requested by enami in ticket #645):
	common/lib/libc/arch/i386/atomic/atomic.S: revision 1.17
	sys/arch/amd64/amd64/spl.S: revision 1.21
	sys/arch/x86/x86/patch.c: revision 1.17
So that profile kernel runs again,
- Adjust the size of functions used to patch.
- Fix the jump offset of mcount call when patching functions.
Approved by Andrew Doran.

So that profile kernel runs again,
- Adjust the size of functions used to patch.
- Fix the jump offset of mcount call when patching functions.

Approved by Andrew Doran.

Sync with HEAD.

Pull up following revision(s) (requested by ad in ticket #471):
	sys/arch/x86/x86/patch.c: revision 1.16
Repair x86_patch to install optimized routines.
Pointed out by enami@.

Repair x86_patch to install optimized routines.
Pointed out by enami@.

Pull up following revision(s) (requested by ad in ticket #343):
	common/lib/libc/arch/i386/atomic/atomic.S: revision 1.14
	sys/arch/x86/include/cpufunc.h: revision 1.9
	sys/arch/x86/x86/identcpu.c: revision 1.12
	sys/arch/x86/x86/cpu.c: revision 1.60
	sys/arch/x86/x86/patch.c: revision 1.15
PR kern/40213 my i386 machine can't boot because of tsc
- Patch in atomic_cas_64() twice. The first patch is early and makes it
  the MP-atomic version available if we have cmpxchg8b. The second patch
  strips the lock prefix if ncpu==1.
- Fix the i486 atomic_cas_64() to not unconditionally enable interrupts.

Sync with HEAD.

Sync with HEAD.

PR kern/40213 my i386 machine can't boot because of tsc

- Patch in atomic_cas_64() twice. The first patch is early and makes it
  the MP-atomic version available if we have cmpxchg8b. The second patch
  strips the lock prefix if ncpu==1.

- Fix the i486 atomic_cas_64() to not unconditionally enable interrupts.

Sync with HEAD.

Sync with HEAD.

Merge in changes between wrstuden-revivesa-base-2 and
wrstuden-revivesa-base-3.

Replace most gcc-specific __attribute__ uses with BSD-style sys/cdef.h
preprocessor macros.

Sync with HEAD.

sync with head.

sync with head.

PR kern/38537 __HAVE_PREEMPTION requires MULTIPROCESSOR

Don't patch out the kernel_lock functions if ncpu == 1. We need it for
preemption, and it's used much less frequently than before.

Remove clause 3 and 4 from TNF licenses

sync with head

sync with HEAD

Sync with HEAD

Sync with HEAD.

Sync with head.

- Make __cpu_simple_lock and similar real functions and patch at runtime.
- Remove old x86 atomic ops.
- Drop text alignment back to 16 on i386 (really, this time).
- Minor cleanup.

64-bit atomic ops for i386.

Sync with HEAD.

sync with head

Sync with HEAD.

Sync with HEAD.

x86_mb_nop is now unused.

Hook in the atomic ops from libkern.

Sync with HEAD.

Sync with HEAD

sync with head.

Sync with HEAD.

When running uniprocessor, patch _kernel_lock() and _kernel_unlock() to
do nothing more than "nop; ret".

Sync with HEAD

Sync with HEAD.

- When computing the TSC frequency, call i8254_delay() and not DELAY().
- Use atomics to adjust the pmap reference count, instead of taking locks.
- Implement I386_{SET,GET}_{FS,GS}BASE, allowing %fs and %gs to be used
  as per-thread registers. This is compatible with FreeBSD.
- Run patches after we have attached CPUs, since we then know if the
  system is uniprocessor or not. Eliminates a lot of #ifdef MULTIPROCESSOR
  and makes running MP kernels on UP systems cheaper.
- Patch out many of the 'lock' prefixes to nops if uniprocessor.
- Do a wbinvd after patching to ensure that the trace/instruction cache
  is up to date.

sync with HEAD

sync with head.

Merge the ppcoea-renovation branch to HEAD.

This branch was a major cleanup and rototill of many of the various OEA
cpu based PPC ports that focused on sharing as much code as possible
between the various ports to eliminate near-identical copies of files in
every tree.  Additionally there is a new PIC system that unifies the
interface to interrupt code for all different OEA ppc arches.  The work
for this branch was done by a variety of people, too long to list here.

TODO:
bebox still needs work to complete the transition to -renovation.
ofppc still needs a bunch of work, which I will be looking at.
ev64260 still needs to be renovated
amigappc was not attempted.

NOTES:
pmppc was removed as an arch, and moved to a evbppc target.

Sync with head.

sync with head.

Sync with HEAD

Sync with HEAD.

x86 changes for pcc and LKMs.

- Replace most inline assembly with proper functions. As a side effect
  this reduces the size of amd64 GENERIC by about 120kB, and i386 by a
  smaller amount. Nearly all of the inlines did something slow, or something
  that does not need to be fast.
- Make curcpu() and curlwp functions proper, unless __GNUC__ && _KERNEL.
  In that case make them inlines. Makes curlwp LKM and preemption safe.
- Make bus_space and bus_dma more LKM friendly.
- Share a few more files between the ports.
- Other minor changes.

sync with head.

Sync with head.

Sync with head.

Update to HEAD.

merge yamt-idlelwp branch.  asked by core@.  some ports still needs work.

from doc/BRANCHES:

	idle lwp, and some changes depending on it.

	1. separate context switching and thread scheduling.
	   (cf. gmcgarry_ctxsw)
	2. implement idle lwp.
	3. clean up related MD/MI interfaces.
	4. make scheduler(s) modular.

New atomic op hot-patches for amd64.

Hot-patch the atomic ops that need it on x86 (32-bit).

Checkpoint:
- Abstract for per-CPU locking of runqueues.
  As a workaround for SCHED_4BSD global runqueue, covered by sched_mutex,
  spc_mutex is a pointer for now. After making SCHED_4BSD runqueues
  per-CPU, it will became a storage mutex.
- suspendsched: Locking is not necessary for cpu_need_resched().
- Remove mutex_spin_exit() prototype in patch.c and LOCK_ASSERT() check
  in runqueue_nextlwp() in sched_4bsd.c to make them compile again.

sync with head.

file patch.c was added on branch yamt-lazymbuf on 2007-02-26 09:08:52 +0000

Merge newlock2 to head.

- Define memory barrier ops in lock_stubs.S.
- If lfence/mfence are available, patch them in at boot.
- Patch to a no-op if !MULTIPROCESSOR. XXX Should be determined at runtime.

- Don't patch stuff if it's a profiling kernel.
- Move Xspllower label to show up better in profile output.

Rename some functions to better describe what they do.

If running on a PPro or later, at boot patch in versions of spllower() and
similar that use cmpxchg8b instead of cli/sti. Cuts the clock cycles for
splx() by a factor of ~6 on the P4, and ~3 on the PIII when bracketed by
serializing instructions (and hopefully more when not).

file patch.c was initially added on branch newlock2.