CVS log for src/sys/arch/x86/x86/x86_tlb.c

Up to [cvs.NetBSD.org] / src / sys / arch / x86 / x86

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

fix triple s typos in comments.

x86: Split most of pmap.h into pmap_private.h or vmparam.h.

This way pmap.h only contains the MD definition of the MI pmap(9)
API, which loads of things in the kernel rely on, so changing x86
pmap internals no longer requires recompiling the entire kernel every
time.

Callers needing these internals must now use machine/pmap_private.h.
Note: This is not x86/pmap_private.h because it contains three parts:

1. CPU-specific (different for i386/amd64) definitions used by...

2. common definitions, including Xenisms like xpmap_ptetomach,
   further used by...

3. more CPU-specific inlines for pmap_pte_* operations

So {amd64,i386}/pmap_private.h defines 1, includes x86/pmap_private.h
for 2, and then defines 3.  Maybe we should split that out into a new
pmap_pte.h to reduce this trouble.

No functional change intended, other than that some .c files must
include machine/pmap_private.h when previously uvm/uvm_pmap.h
polluted the namespace with pmap internals.

Note: This migrates part of i386/pmap.h into i386/vmparam.h --
specifically the parts that are needed for several constants defined
in vmparam.h:

VM_MAXUSER_ADDRESS
VM_MAX_ADDRESS
VM_MAX_KERNEL_ADDRESS
VM_MIN_KERNEL_ADDRESS

Since i386 needs PDP_SIZE in vmparam.h, I added it there on amd64
too, just to keep things parallel.

KNF. No functional change.

Merge changes from current as of 20200406

x86 pmap:

- Give pmap_remove_all() its own version of pmap_remove_ptes() that on native
  x86 does the bare minimum needed to clear out PTPs.  Cuts ~4% sys time on
  'build.sh release' for me.

- pmap_sync_pv(): there's no need to issue a redundant TLB shootdown.  The
  caller waits for the competing operation to finish.

- Bring 'options TLBSTATS' up to date.

Sync with head.

Adjustment to previous: TP_SET_DONE() was wiping out the VA to shoot,
instead of ORing the flag into the array element.  This caused the CPU
initiating the shootdown to occasionally miss an INVLPG.

add relaxed atomics, ok ad@ riastradh@

Sync with head.

Push the INVLPG limit for shootdowns up to 16 (for UBC).

x86 pmap:

- It turns out that every page the pmap frees is necessarily zeroed.  Tell
  the VM system about this and use the pmap as a source of pre-zeroed pages.

- Redo deferred freeing of PTPs more elegantly, including the integration with
  pmap_remove_all().  This fixes problems with nvmm, and possibly also a crash
  discovered during fuzzing.

Reported-by: syzbot+a97186518c84f1d85c0c@syzkaller.appspotmail.com

Align the TLB packet precisely on the stack, and do 7 INVLPG since it's
what fits in a single line.

Fix tyo in comment

Fix a hard hang with Xen MP.

Minor correction to previous.

x86 TLB shootdown IPI changes:

- Shave some time off processing.
- Reduce cacheline/bus traffic on systems with many CPUs.
- Reduce time spent at IPL_VM.

Sync with HEAD

Remove 'ci_svs_kpdirpa', unused. While here fix a few comments here and
there, reduces a future diff.

Rename the PTE bits.

Another locking issue in NVMM: the {svm,vmx}_tlb_flush functions take VCPU
mutexes which can sleep, but their context does not allow it.

Rewrite the TLB handling code to fix that. It becomes a bit complex. In
short, we use a per-VM generation number, which we increase on each TLB
flush, before sending a broadcast IPI to everybody. The IPIs cause a
#VMEXIT of each VCPU, and each VCPU Loop will synchronize the per-VM gen
with a per-VCPU copy, and apply the flushes as neededi lazily.

The behavior differs between AMD and Intel; in short, on Intel we don't
flush the hTLB (EPT cache) if a context switch of a VCPU occurs, so now,
we need to maintain a kcpuset to know which VCPU's hTLBs are active on
which hCPU. This creates some redundancy on Intel, ie there are cases
where we flush the hTLB several times unnecessarily; but hTLB flushes are
very rare, so there is no real performance regression.

The thing is lock-less and non-blocking, so it solves our problem.

We reorganise definitions for XEN source support as follows:

XEN - common sources required for baseline XEN support.
XENPV - sources required for support of XEN in PV mode.
XENPVHVM - sources required for support for XEN in HVM mode.
XENPVH - sources required for support for XEN in PVH mode.

Synch with HEAD

Flush the host TLB too when dealing with a guest pmap. The pmap is not
active on the host so the pages aren't cached; but the recursive PTE
entries may have been cached by our pmap code.

Sync with HEAD, resolve a couple of conflicts

Add two pmap fields, will be used by NVMM.

Sync with HEAD

remove more vestiges of uvm_emap_*(), to fix x86 kernel linking

rename sys/arch/x86/x86/pmap_tlb.c to sys/arch/x86/x86/x86_tlb.c, so that
x86 can eventually use uvm/pmap/pmap_tlb.c; step to future PCID support