CVS log for src/sys/arch/x86/x86/bus_dma.c

Up to [cvs.NetBSD.org] / src / sys / arch / x86 / x86

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

Pull up following revision(s) (requested by rin in ticket #928):

	sys/external/bsd/drm2/dist/drm/drm_gem.c: revision 1.25
	sys/external/bsd/drm2/dist/drm/radeon/radeon_ci_dpm.c: revision 1.7
	sys/external/bsd/drm2/dist/drm/nouveau/nvkm/engine/device/priv.h: revision 1.4
	sys/external/bsd/drm2/dist/drm/nouveau/nvkm/engine/device/nouveau_nvkm_engine_device_acpi.c: revision 1.4
	sys/external/bsd/drm2/dist/drm/i915/display/intel_opregion.h: revision 1.6
	sys/external/bsd/drm2/dist/drm/i915/i915_drv.h: revision 1.49
	sys/external/bsd/drm2/include/linux/mxm-wmi.h: revision 1.1
	sys/external/bsd/drm2/dist/drm/nouveau/nvkm/subdev/pci/nouveau_nvkm_subdev_pci_pcie.c: revision 1.4
	sys/external/bsd/drm2/dist/drm/nouveau/nvkm/engine/device/nouveau_nvkm_engine_device_base.c: revision 1.13
	sys/external/bsd/common/include/linux/bitops.h: revision 1.17
	sys/external/bsd/drm2/nouveau/files.nouveau: revision 1.40
	sys/external/bsd/drm2/linux/linux_pci.c: revision 1.30
	sys/external/bsd/drm2/dist/drm/radeon/radeon_drv.h: revision 1.5
	sys/external/bsd/drm2/dist/drm/nouveau/nvkm/subdev/pci/nouveau_nvkm_subdev_pci_pcie.c: revision 1.5
	sys/external/bsd/drm2/dist/drm/nouveau/nvkm/subdev/mxm/nouveau_nvkm_subdev_mxm_base.c: revision 1.5
	sys/external/bsd/drm2/dist/drm/amd/amdgpu/amdgpu_gart.c: revision 1.12
	sys/external/bsd/drm2/dist/drm/radeon/radeon_rv770.c: revision 1.3
	sys/external/bsd/drm2/dist/drm/nouveau/nvkm/engine/disp/nouveau_nvkm_engine_disp_sorgm200.c: revision 1.3
	sys/external/bsd/common/include/linux/printk.h: revision 1.14
	sys/external/bsd/drm2/dist/drm/nouveau/nvkm/subdev/instmem/nouveau_nvkm_subdev_instmem_gk20a.c: revision 1.10
	sys/external/bsd/drm2/dist/drm/amd/amdgpu/amdgpu_vi.c: revision 1.4
	sys/external/bsd/drm2/include/linux/acpi.h: revision 1.11
	sys/external/bsd/drm2/drm/drm_cdevsw.c: revision 1.31
	sys/external/bsd/drm2/dist/drm/radeon/radeon_si.c: revision 1.6
	sys/external/bsd/drm2/dist/drm/nouveau/nouveau_acpi.c: revision 1.5
	sys/external/bsd/drm2/dist/drm/i915/display/intel_acpi.h: revision 1.5
	sys/external/bsd/drm2/include/acpi/video.h: revision 1.3
	sys/external/bsd/drm2/dist/drm/radeon/radeon_evergreen.c: revision 1.6
	sys/external/bsd/drm2/dist/drm/nouveau/nouveau_acpi.h: revision 1.4
	sys/arch/sparc64/include/pci_machdep.h: revision 1.31
	sys/arch/sparc64/dev/pci_machdep.c: revision 1.83
	sys/external/bsd/drm2/include/linux/kref.h: revision 1.14
	sys/external/bsd/drm2/dist/drm/nouveau/nvkm/engine/device/nouveau_nvkm_engine_device_pci.c: revision 1.12
	sys/external/bsd/drm2/linux/linux_dma_buf.c: revision 1.17
	sys/external/bsd/drm2/dist/drm/nouveau/nvkm/subdev/bios/nouveau_nvkm_subdev_bios_shadowacpi.c: revision 1.4
	sys/external/bsd/drm2/drm/drm_module.c: revision 1.32
	sys/external/bsd/drm2/dist/drm/i915/i915_gem.h: revision 1.8
	sys/external/bsd/drm2/dist/drm/i915/gem/i915_gem_dmabuf.c: revision 1.7
	sys/external/bsd/drm2/include/linux/smp.h: revision 1.5
	sys/external/bsd/drm2/dist/drm/amd/amdgpu/amdgpu_si.c: revision 1.5
	sys/external/bsd/drm2/dist/drm/amd/amdgpu/amdgpu_device.c: revision 1.20
	sys/arch/x86/x86/bus_dma.c: revision 1.91
	sys/external/bsd/drm2/radeon/files.radeon: revision 1.40
	sys/external/bsd/drm2/include/acpi/acpi_bus.h: revision 1.1
	sys/external/bsd/drm2/dist/drm/nouveau/nouveau_drv.h: revision 1.5
	sys/external/bsd/drm2/dist/drm/amd/amdgpu/amdgpu_device.c: revision 1.21
	sys/external/bsd/common/include/asm/barrier.h: revision 1.20
	sys/external/bsd/drm2/include/linux/nbsd-namespace-acpi.h: revision 1.2
	sys/external/bsd/common/include/asm/barrier.h: revision 1.21
	sys/modules/drmkms/drmkms_pci.h: revision 1.1
	sys/external/bsd/drm2/dist/drm/drm_dp_helper.c: revision 1.17
	sys/external/bsd/drm2/radeon/radeon_pci.c: revision 1.23
	sys/external/bsd/drm2/linux/linux_xa.c: revision 1.4
	sys/external/bsd/drm2/ttm/ttm_bo_vm.c: revision 1.23
	sys/external/bsd/drm2/ttm/ttm_bo_vm.c: revision 1.24
	sys/external/bsd/drm2/ttm/ttm_bo_vm.c: revision 1.25
	sys/external/bsd/drm2/linux/linux_pci.c: revision 1.26
	sys/dev/pci/pcivar.h: revision 1.120
	sys/arch/xen/include/pci_machdep.h: revision 1.24
	sys/external/bsd/drm2/ttm/ttm_bo_vm.c: revision 1.26
	sys/external/bsd/drm2/linux/linux_pci.c: revision 1.27
	sys/external/bsd/drm2/ttm/ttm_bo_vm.c: revision 1.27
	sys/external/bsd/drm2/linux/linux_pci.c: revision 1.28
	sys/external/bsd/drm2/dist/drm/radeon/radeon_cik.c: revision 1.8
	sys/external/bsd/drm2/ttm/ttm_bo_vm.c: revision 1.28
	sys/external/bsd/drm2/linux/linux_pci.c: revision 1.29
	sys/external/bsd/drm2/include/linux/pci.h: revision 1.57
	sys/external/bsd/drm2/include/linux/pci.h: revision 1.58
	sys/external/bsd/drm2/dist/drm/radeon/radeon_acpi.c: revision 1.5
	sys/external/bsd/drm2/dist/drm/nouveau/nouveau_display.c: revision 1.6
	sys/external/bsd/drm2/dist/drm/amd/powerplay/hwmgr/amdgpu_hwmgr.c: revision 1.3
	sys/external/bsd/drm2/dist/drm/amd/display/dc/core/amdgpu_dc_stream.c: revision 1.3
	share/man/man9/bus_dma.9: revision 1.69
	sys/external/bsd/drm2/drm/drm_gem_cma_helper.c: revision 1.15
	sys/external/bsd/drm2/dist/drm/radeon/radeon_acpi.c: revision 1.6
	sys/external/bsd/drm2/dist/drm/radeon/radeon.h: revision 1.12
	sys/external/bsd/drm2/dist/drm/amd/amdgpu/amdgpu_cik.c: revision 1.7
	sys/dev/acpi/acpi_mcfg.c: revision 1.29
	sys/external/bsd/drm2/dist/drm/amd/amdgpu/amdgpu_acpi.c: revision 1.6
	sys/external/bsd/drm2/dist/drm/radeon/radeon_r600.c: revision 1.7
	sys/external/bsd/drm2/dist/drm/radeon/radeon_bios.c: revision 1.13
	sys/modules/amdgpu/Makefile: revision 1.9
	sys/external/bsd/drm2/dist/drm/radeon/radeon_bios.c: revision 1.14
	sys/external/bsd/common/linux/linux_tasklet.c: revision 1.12
	sys/external/bsd/drm2/dist/drm/nouveau/include/nvkm/core/device.h: revision 1.10
	sys/external/bsd/drm2/dist/drm/i915/gem/i915_gem_mman.c: revision 1.23
	sys/external/bsd/drm2/dist/drm/amd/amdgpu/amdgpu.h: revision 1.9
	sys/external/bsd/drm2/include/linux/interval_tree.h: revision 1.14
	sys/external/bsd/drm2/dist/drm/i915/gem/i915_gem_mman.c: revision 1.26
	sys/external/bsd/drm2/dist/drm/amd/powerplay/hwmgr/amdgpu_smu7_hwmgr.c: revision 1.5
	sys/dev/pci/pci.c: revision 1.168
	sys/external/bsd/drm2/dist/drm/i915/gem/i915_gem_mman.c: revision 1.27
	sys/external/bsd/drm2/dist/drm/radeon/radeon_si_dpm.c: revision 1.9
	sys/external/bsd/drm2/pci/files.drmkms_pci: revision 1.18
	sys/external/bsd/drm2/linux/linux_sync_file.c: revision 1.3
	sys/external/bsd/drm2/amdgpu/files.amdgpu: revision 1.31
	sys/external/bsd/drm2/dist/drm/nouveau/nvkm/engine/device/nouveau_nvkm_engine_device_tegra.c: revision 1.4
	sys/external/bsd/drm2/dist/drm/drm_gem.c: revision 1.24
	sys/arch/xen/xen/xpci_xenbus.c: revision 1.29

drm: Eliminate __HAVE_ATOMIC_AS_MEMBAR conditionals.
Discussed on tech-kern:
https://mail-index.netbsd.org/tech-kern/2023/02/23/msg028729.html

linux asm/barrier.h: Fix !MULTIPROCESSOR build.

remove "nouveau" from a comment.  noted by jmcneill.

drm: KASSERT(A && B) -> KASSERT(A); KASSERT(B)
comment a function that has a clear overbounds read but it isn't used.
found by GCC 12.

nix the NetBSD specific GEM_BUG_ON().
avoids GCC 12 warnings, and matches upstream closer.
avoid uninitialised variable usage in drm_gem_cma_create_internal().
in the case nothing has returned 'error', 'nsegs' and the dma info
are (potentially) uninitialised, so consider this an error.
found by GCC 12.

avoid a GCC 12 warning.
there's a 1-element long array and a loop conditional that tries to see
if indexes for it are not identical.  as these indexes will always both
be 0, the only valid index, the condition is always false.  GCC 12
triggers a strange warning on this code that can never run (see below),
so simply assert the array size is 1 and comment the rest.
amdgpu_dc_stream.c:470:55: error: array subscript [0, 0] is outside array bounds of 'struct dc_writeback_info[1]' [-Werror=array-bounds]
  470 |                                 stream->writeback_info[j] = stream->writeback_info[i];

convert a KASSERT() into an if () panic() sequence to appease GCC 12.
OK riastradh@.

drm: Fix conditionals around drmkms_pci and agp.
Kernel should build now with all pci drm drivers stripped out but
DRM_LEGACY still enabled.  (Might not be very useful, but it'll
build.  Maybe we should also have DRM_LEGACY_PCI so those drivers can
be modloaded later.)

drmkms: Fix module build.
avoid an unlikely array bounds issue picked up by GCC 12.
nvkm_pcie_speed() can return -1, which is then used as an array index,
so make this default return PCIe 1.0 speeds.

drm: enable almost all PCIe functionality
linux_pci.c revisions 1.24 and 1.25 implemented most of the remaining
missing PCIe backends, but only enabled them for some amdgpu portions.
this enables all code marked with "XXX amdgpu pcie", "XXX radeon pcie",
and "XXX pcie speed".  for most of it, simply removing #ifndefs __NetBSD__
to enable compliation was required, once the new "bus->max_bus_speed"
member was added to struct pci_bus.  add an "always fails" backend for
pci_enable_atomic_ops_to_root() which seems to only be necessary
for virtual GPU functionality (and could be implemented if needed.)
tested on radeon 5450, 7750, R7 240 [radeon], and RX 550 [amdgpu], and
nvidia 750 and 1030 [nouveau].
this still does not quite work on nvidia cards.  there are two problems
that remain:
- the call to set the link speed is skipped because the speed is set
  to the default value of "-1".  nvkm_pcie_set_link() will actually
  determine the right value for this and for some cards, calling this
  function if the current speed is -1 helps set the link speed.  it
  may be that on linux other paths we don't have enabled properly
  would set this (there's one via debugfs, and a jetson specific one,
  though perhaps setting either AC or DC speed values as boot options
  (after hooking up these for netbsd) would currently work.
- worse, cards newer than kepler - geforce 900, 1000, and newer, are
  all lacking the backing support to set pcie link speed.  the GT 1030
  card i have been testing with remains at pcie 1.0.

radeon: fix and enable ACPI methods for getting ROM BIOS
The hacky way of getting the BIOS mapped only works on x86. ACPI
should be preferred if available. Makes BIOS reading though VFCT
work on aarch64 with EDK2. (But only if EDK2 has POSTed the GPU.)
XXX amdgpu should get the same treatment.

drm: put_cpu() should enable preemption, not disable it again

drm(4): make pr_debug equivalent to aprint_debug
significantly reduces the default spam from amdgpu(4).

drm: Set CONFIG_ACPI in linux/acpi.h and make it build.

Leave a little ACPI-related functionality disabled for now, like
getting EDID out of ACPI -- needs a bit more work to make this work,
and I don't have hardware to work on that.
Should help with failures of the forms:
- unable to locate a BIOS ROM
- bios: unable to locate usable image
on various machines.

radeon_acpi.c: ifdef out unused function on NetBSD.
Should fix syzkaller build.

drm(4): Fix st_rdev in stat.
dminor->index already has the 64*type adjustment, as allocated in
drm_minor_alloc.
PR kern/58180

linux_sync_file: Fix missing init/fini steps.
Noted by rjs@.
PR kern/58210

ttm: Sync ttm_bo_uvm_fault_idle better with Linux.
PR xsrc/58133
ttm: Undo mistake in previous.

PR xsrc/58133
linux: Add a few more cases to pci_get_class.
Should fix crash on boot with amdgpu now that the ACPI business is
enabled.

i915: Fix dmabuf mmap object.

drm: Fix missing bounds checks in dma buf mmap.

drm_gem.c: Fix sense of assertion.
This is the opposite of WARN_ON.
Noted by rjs@.

drm_gem.c: Enable drm_gem_fence_array_add now that we emulate xa.
linux_xa: Delete and replace collision in xa_store as intended.
Don't free the colliding node that's still in the tree.
Noted by rjs@.

i915_gem_mman.c: Apply mmap types via pmap flags.
This way, userland gets buffers mapped write-combining or uncached as
needed.
PR xsrc/58307

x86: Teach bus_dmamem_map about BUS_DMA_PREFETCHABLE.
PR port-amd64/58308

bus_dma(9): Document BUS_DMA_PREFETCHABLE.
Like BUS_DMA_NOCACHE.  Doesn't absolve you of the need for
bus_dmamap_sync, but if you later pass the vaddr to bus_dmamap_load,
the DMA map might notice the mapping is write-combining and use this
to make bus_dmamap_sync cheaper.
PR kern/58309

nouveau_nvkm_subdev_instmem_gk20a.c: Use BUS_DMA_PREFETCHABLE.
Matches Linux's pgprot_writecombine.
Unclear where the appropriate bus_dmamap_sync happens, or is supposed
to happen -- not using it would be wrong, but asking for a
prefetchable mapping may paper over symptoms, at least!

ttm: Sync more with Linux.
Add the original copyright and attribution since this is now,
intentionally, a modified copy of the original and not just roughly
the same algorithm.

ttm: Respect PGO_ALLPAGES.
Not sure this is useful but it reduces XXX's and makes this match
udv_fault better so it's easier to understand.

ttm: Sync cacheability flag logic with Linux.

ttm: Add XXX about readahead fault failures.

pci: Pass cookie through pci_find_device, pci_enumerate_bus, take 2.
New functions pci_find_device1 and pci_enumerate_bus1 have the cookie
argument.  Existing symbols pci_find_device and pci_enumerate_bus are
now wrappers for the cookieless version.
This will allow pci_find_device callers to pass a cookie through to
the match function so they can keep state or pass in extra parameters
like b/d/f numbers, which will allow us to nix some horrible kludges
in the Linux PCI API emulation for drm (and, perhaps, Intel wifi).
This change drops the symbol pci_probe_device, in favour of a new
pci_probe_device1 with the cookie argument.  But I don't think that
requires a revbump because it's only called by MD pci_enumerate_bus1
implementations, which don't live in modules anyway.
Take 2: Make sure to handle NULL match function.
linux_pci: Nix pci enumeration kludges.
Now that we can pass a cookie through, this stuff will be a little
less fragile.

i915: Omit needless i915_gem_object_pin/unpin_pages cycle in fault.
vm_fault_cpu and vm_fault_gtt, called by i915_gem_fault, already do
the pinning and unpinning internally, so there is no need for
i915_gem_fault to do it.
No functional change intended, except that the transient pin count
will be one lower than before during the fault routine (but it will
still be positive).

i915: Match Linux fault routine return code actions.
Omit needless EINTR interception -- this is now handled by
i915_error_to_vmf_fault.
Earlier revert was over a false alarm -- bisection shows the new
warnings arose from linux_pci.c 1.29 here:
https://mail-index.netbsd.org/source-changes/2024/06/23/msg151929.html

linux_pci: Fix shifto in pci_get_class.
It looks like Linux's pci_get_class also matches the interface part
of the PCI class register (but not the revision part), and I hadn't
noticed that in the previous shim structured differently.

With GCC12 kernel ALL/amd64 triggers "'sor' may be used uninitialized".
If "sublinks & 3" is zero GCC is right and sor[1] may be returned unitialized.
Fix by initializing "sor" to zero to return -1 instead of uninitialized value.
Ok: Taylor R Campbell <riastradh@>

amdgpu: Map BAR 2, not BAR 5, on pre-bonaire chips.
PR kern/58384

amdgpu: Map consecutive pages, not the same one over and over again.
PR kern/58385

linux/bitops: Fix overestimate for BITS_TO_LONGS(9)
Fortunately, this seems harmless except for allocating
excessive buffer memory.
Pointed out by nonaka@, OK riastradh@.

Pull up following revision(s) (requested by rin in ticket #881):

	sys/arch/x86/x86/bus_dma.c: revision 1.90

x86/bus_dma.c: Sprinkle KASSERTMSG.

x86: Teach bus_dmamem_map about BUS_DMA_PREFETCHABLE.

PR port-amd64/58308

x86/bus_dma.c: Sprinkle KASSERTMSG.

x86: Split most of pmap.h into pmap_private.h or vmparam.h.

This way pmap.h only contains the MD definition of the MI pmap(9)
API, which loads of things in the kernel rely on, so changing x86
pmap internals no longer requires recompiling the entire kernel every
time.

Callers needing these internals must now use machine/pmap_private.h.
Note: This is not x86/pmap_private.h because it contains three parts:

1. CPU-specific (different for i386/amd64) definitions used by...

2. common definitions, including Xenisms like xpmap_ptetomach,
   further used by...

3. more CPU-specific inlines for pmap_pte_* operations

So {amd64,i386}/pmap_private.h defines 1, includes x86/pmap_private.h
for 2, and then defines 3.  Maybe we should split that out into a new
pmap_pte.h to reduce this trouble.

No functional change intended, other than that some .c files must
include machine/pmap_private.h when previously uvm/uvm_pmap.h
polluted the namespace with pmap internals.

Note: This migrates part of i386/pmap.h into i386/vmparam.h --
specifically the parts that are needed for several constants defined
in vmparam.h:

VM_MAXUSER_ADDRESS
VM_MAX_ADDRESS
VM_MAX_KERNEL_ADDRESS
VM_MIN_KERNEL_ADDRESS

Since i386 needs PDP_SIZE in vmparam.h, I added it there on amd64
too, just to keep things parallel.

Fix an inverted KASSERTMSG test from the #ifdef DIAGNOSTIC panic -> KASSERT
conversion.

x86/bus_dma: #ifdef DIAGNOSTIC panic -> KASSERT

While here, use some better types and avoid integer overflow in the
diagnostic tests.

No functional change intended except in the case of bugs anyway.

x86: Adjust fences issued in bus_dmamap_sync after bouncing.

And expand the comment on the lfence for POSTREAD before bouncing.

Net change:

op                      before bounce       after bounce
                                        old             new
PREREAD                 nop             lfence          sfence
PREWRITE                nop             mfence          sfence
PREREAD|PREWRITE        nop             mfence          sfence
POSTREAD                lfence          lfence          nop[*]
POSTWRITE               nop             mfence          nop
POSTREAD|POSTWRITE      lfence          mfence          nop[*]

The case of PREREAD is as follows:

1. loads and stores before DMA buffer may be allocated for the purpose
2. bus_dmamap_sync(BUS_DMASYNC_PREREAD)
3. store to register or DMA descriptor to trigger DMA

The register or DMA descriptor may be in any type of memory (or I/O).

lfence at (2) is _not enough_ to ensure stores at (1) have completed
before the store in (3) in case the register or DMA descriptor lives
in wc/wc+ memory, or the store to it is non-temporal: in that case,
it may executed early before all the stores in (1) have completed.

On the other hand, lfence at (2) is _not needed_ to ensure loads in
(1) have completed before the store in (3), because x86 never
reorders load;store to store;load.  So we may need to enforce
store/store ordering, but not any other ordering, hence sfence.

The case of PREWRITE is as follows:

1. stores to DMA buffer (and loads from it, before allocated)
2. bus_dmamap_sync(BUS_DMASYNC_PREWRITE)
3. store to register or DMA descriptor to trigger DMA

Ensuring prior loads have completed is not necessary because x86
never reorders load;store to store;load (and in any case, the device
isn't changing the DMA buffer, so it's safe to read over and over
again).  But we must ensure the stores in (1) have completed before
the store in (3).  So we need sfence, in case either the DMA buffer
or the register or the DMA descriptor is in wc/wc+ memory or either
store is non-temporal.  But we don't need mfence.

The case of POSTREAD is as follows:

1. load from register or DMA descriptor notifying DMA completion
2. bus_dmamap_sync(BUS_DMASYNC_POSTREAD)
   (a) lfence [*]
   (b) if bouncing, memcpy(userbuf, bouncebuf, ...)
   (c) ???
3. loads from DMA buffer to use data, and stores to reuse buffer

This certainly needs an lfence to prevent the loads at (3) from being
executed early -- but bus_dmamap_sync already issues lfence in that
case at 2(a), before it conditionally loads from the bounce buffer
into the user's buffer.  So we don't need any _additional_ fence
_after_ bouncing at 2(c).

The case of POSTWRITE is as follows:

1. load from register or DMA descriptor notifying DMA completion
2. bus_dmamap_sync(BUS_DMASYNC_POSTWRITE)
3. loads and stores to reuse buffer

Stores at (3) will never be executed early because x86 never reorders
load;store to store;load for any memory types.  Loads at (3) are
harmless because the device isn't changing the buffer -- it's
supposed to be fixed from the time of PREWRITE to the time of
POSTWRITE as far as the CPU can witness.

Proposed on port-amd64 last month:

https://mail-index.netbsd.org/port-amd64/2022/07/16/msg003593.html

Reference:

AMD64 Architecture Programmer's Manual, Volume 2: System Programming,
24593--Rev. 3.38--November 2021, Sec. 7.4.2 Memory Barrier Interaction
with Memory Types, Table 7-3, p. 196.
https://www.amd.com/system/files/TechDocs/24593.pdf

x86: Move lfence into _bus_dmamap_sync and comment it.

No functional change intended.  This just keeps the bus_dma_* and
_bus_dma_* functions organized more consistently.

Ensure bus_dmatag_subregion is called with an inclusive max_addr
everywhere.

KNF. No functional change.

Mostly merge changes from HEAD upto 20200411

Merge changes from current as of 20200406

- Hide the details of SPCF_SHOULDYIELD and related behind a couple of small
  functions: preempt_point() and preempt_needed().

- preempt(): if the LWP has exceeded its timeslice in kernel, strip it of
  any priority boost gained earlier from blocking.

Add support for Kernel Memory Sanitizer (kMSan). It detects uninitialized
memory used by the kernel at run time, and just like kASan and kCSan, it
is an excellent feature. It has already detected 38 uninitialized variables
in the kernel during my testing, which I have since discreetly fixed.

We use two shadows:
 - "shad", to track uninitialized memory with a bit granularity (1:1).
   Each bit set to 1 in the shad corresponds to one uninitialized bit of
   real kernel memory.
 - "orig", to track the origin of the memory with a 4-byte granularity
   (1:1). Each uint32_t cell in the orig indicates the origin of the
   associated uint32_t of real kernel memory.

The memory consumption of these shadows is consequent, so at least 4GB of
RAM is recommended to run kMSan.

The compiler inserts calls to specific __msan_* functions on each memory
access, to manage both the shad and the orig and detect uninitialized
memory accesses that change the execution flow (like an "if" on an
uninitialized variable).

We mark as uninit several types of memory buffers (stack, pools, kmem,
malloc, uvm_km), and check each buffer passed to copyout, copyoutstr,
bwrite, if_transmit_lock and DMA operations, to detect uninitialized memory
that leaves the system. This allows us to detect kernel info leaks in a way
that is more efficient and also more user-friendly than KLEAK.

Contrary to kASan, kMSan requires comprehensive coverage, ie we cannot
tolerate having one non-instrumented function, because this could cause
false positives. kMSan cannot instrument ASM functions, so I converted
most of them to __asm__ inlines, which kMSan is able to instrument. Those
that remain receive special treatment.

Contrary to kASan again, kMSan uses a TLS, so we must context-switch this
TLS during interrupts. We use different contexts depending on the interrupt
level.

The orig tracks precisely the origin of a buffer. We use a special encoding
for the orig values, and pack together in each uint32_t cell of the orig:
 - a code designating the type of memory (Stack, Pool, etc), and
 - a compressed pointer, which points either (1) to a string containing
   the name of the variable associated with the cell, or (2) to an area
   in the kernel .text section which we resolve to a symbol name + offset.

This encoding allows us not to consume extra memory for associating
information with each cell, and produces a precise output, that can tell
for example the name of an uninitialized variable on the stack, the
function in which it was pushed on the stack, and the function where we
accessed this uninitialized variable.

kMSan is available with LLVM, but not with GCC.

The code is organized in a way that is similar to kASan and kCSan, so it
means that other architectures than amd64 can be supported.

Add DMA instrumentation in KASAN. We note the original buffer and length in
the map, and check the buffer on each bus_dmamap_sync. This allows us to
find DMA buffer overflows and UAFs, which couldn't be found before because
the device accesses to memory are outside of KASAN's control.

KASSERT() -> KASSERTMSG() message in _bus_dmamem_alloc_range().

Sync with HEAD

Rename the PTE bits.

update from HEAD

Sync with HEAD

modify code handling mismatch of nsegs in _bus_dmamem_alloc_range() to
a KASSERT() - plain return leaks memory, and this condition should
never trigger unless there is bug in uvm_pglistalloc(), so it seems
to be waste to check this

other ports usually simply do not check this, with exception of arm,
which does even full cleanup (checks it and calls uvm_pglistfree())

remove checks for failure after memory allocation calls that cannot fail:

  kmem_alloc() with KM_SLEEP
  kmem_zalloc() with KM_SLEEP
  percpu_alloc()
  pserialize_create()
  psref_class_create()

all of these paths include an assertion that the allocation has not failed,
so callers should not assert that again.

Sync with HEAD

Sync with HEAD.  (Note that most of these changes are simply $NetBSD$
tag issues.)

Update the dmamp argument only when the allocation succeeded.

Sync with HEAD (as of 26th Dec)

Pull up following revision(s) (requested by christos in ticket #1339):
	sys/arch/x86/x86/bus_dma.c: revision 1.72
	sys/arch/x86/x86/bus_dma.c: revision 1.73
	sys/arch/x86/x86/bus_dma.c: revision 1.74
- If we succeeded allocating a buffer that did not need bouncing before, but
  the buffer in the previous mapping did, clear the bounce bit. Fixes the
  ld_virtio.c bug with machines 8GB and dd if=/dev/zero of=crash bs=1g count=4.
- Allocate with M_ZERO instead of doing memset
- The panic string can take a format, use it.
- When checking for the bounce buffer boundary check addr + len < limit, not
  addr < limit.
make sure we have a cookie before we try to clear it.
fix operator precedence.

Pull up following revision(s) (requested by christos in ticket #1339):
	sys/arch/x86/x86/bus_dma.c: revision 1.72
	sys/arch/x86/x86/bus_dma.c: revision 1.73
	sys/arch/x86/x86/bus_dma.c: revision 1.74
- If we succeeded allocating a buffer that did not need bouncing before, but
  the buffer in the previous mapping did, clear the bounce bit. Fixes the
  ld_virtio.c bug with machines 8GB and dd if=/dev/zero of=crash bs=1g count=4.
- Allocate with M_ZERO instead of doing memset
- The panic string can take a format, use it.
- When checking for the bounce buffer boundary check addr + len < limit, not
  addr < limit.
make sure we have a cookie before we try to clear it.
fix operator precedence.

Pull up following revision(s) (requested by christos in ticket #1339):
	sys/arch/x86/x86/bus_dma.c: revision 1.72
	sys/arch/x86/x86/bus_dma.c: revision 1.73
	sys/arch/x86/x86/bus_dma.c: revision 1.74
- If we succeeded allocating a buffer that did not need bouncing before, but
  the buffer in the previous mapping did, clear the bounce bit. Fixes the
  ld_virtio.c bug with machines 8GB and dd if=/dev/zero of=crash bs=1g count=4.
- Allocate with M_ZERO instead of doing memset
- The panic string can take a format, use it.
- When checking for the bounce buffer boundary check addr + len < limit, not
  addr < limit.
make sure we have a cookie before we try to clear it.
fix operator precedence.

Pull up following revision(s) (requested by christos in ticket #1011):
	sys/arch/x86/x86/bus_dma.c: revision 1.72
	sys/arch/x86/x86/bus_dma.c: revision 1.73
	sys/arch/x86/x86/bus_dma.c: revision 1.74
- If we succeeded allocating a buffer that did not need bouncing before, but
  the buffer in the previous mapping did, clear the bounce bit. Fixes the
  ld_virtio.c bug with machines 8GB and dd if=/dev/zero of=crash bs=1g count=4.
- Allocate with M_ZERO instead of doing memset
- The panic string can take a format, use it.
- When checking for the bounce buffer boundary check addr + len < limit, not
  addr < limit.
make sure we have a cookie before we try to clear it.
fix operator precedence.

Pull up following revision(s) (requested by christos in ticket #1011):
	sys/arch/x86/x86/bus_dma.c: revision 1.72
	sys/arch/x86/x86/bus_dma.c: revision 1.73
	sys/arch/x86/x86/bus_dma.c: revision 1.74
- If we succeeded allocating a buffer that did not need bouncing before, but
  the buffer in the previous mapping did, clear the bounce bit. Fixes the
  ld_virtio.c bug with machines 8GB and dd if=/dev/zero of=crash bs=1g count=4.
- Allocate with M_ZERO instead of doing memset
- The panic string can take a format, use it.
- When checking for the bounce buffer boundary check addr + len < limit, not
  addr < limit.
make sure we have a cookie before we try to clear it.
fix operator precedence.

fix operator precedence.

make sure we have a cookie before we try to clear it.

- If we succeeded allocating a buffer that did not need bouncing before, but
  the buffer in the previous mapping did, clear the bounce bit. Fixes the
  ld_virtio.c bug with machines 8GB and dd if=/dev/zero of=crash bs=1g count=4.
- Allocate with M_ZERO instead of doing memset
- The panic string can take a format, use it.
- When checking for the bounce buffer boundary check addr + len < limit, not
  addr < limit.

Rebase to HEAD as of a few days ago.

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs.  ("Protocol error: too many arguments")

sync with head

use __func__

sync with head

make a diagnostic message more informative.

resync with head

sync with (a bit old) head

#ifdef - #endif-ed. NMCA, NISA, NNPX, NIOAPIC, LAPIC, MPBIOS and MULTIPROCESSOR.

Both bdt_ov->ov_dmamap_sync() and bus_dmamap_sync() return void,
so don't write return bdt_ov->ov_dmamap_sync(). Pointed out by njoly@

Cosmetic: join some if-statements, remove superfluous parentheses.  No
change in the generated assembly.

After bouncing in bus_dmamap_load{,_mbuf,_uio}, call bus_dmamap_load(9)
instead of _bus_dmamap_load() so that a bus_dmamap_load(9) override has
a shot at loading the map.

XXX Perhaps bounce buffers should be rewritten in terms of bus_dma(9)
XXX overrides.

In bus_dma_tag_create(9), copy important properties (e.g., bounce
parameters) from the parent tag.

In bus_dma_tag_create(), increase the reference count on a parent
bus_dma_tag_t (if applicable), and decrease the reference count in
bus_dma_tag_destroy().

Don't let bus_dmatag_destroy(9) destroy an overridden bus_dma_tag_t.

Add an untested implementation of bus_dmamap_load_raw(9).

Instead of declaring _bus_dmamap_load_busaddr() static inline, make
it static and let the compiler decide about inlining.  This reduces
the code size on both amd64 and i386, and the smaller code is probably
faster code.

In _bus_dmamap_load_busaddr(), change sgsize from an int to a bus_size_t.

Make the 'size' argument of _bus_dmamap_load_busaddr() a bus_size_t for
consistency's sake.

For consistency, call a bus_dma_tag_t bdt instead of bst.  No functional
change intended.

Add bus_dma overrides. From dyoung

Sync with HEAD. Most notably: uvm/pmap work done by rmind@, and MP Xen
work of cherry@.

No regression observed on suspend/restore.

In _bus_dmamap_load_busaddr(), just return 0 instead of assigning an
intermediate variable (int error = 0;) and returning that (return
error;).

#include <sys/bus.h> instead of <machine/bus.h>.

sync with head

Sync with HEAD

Pull up following revision(s) (requested by bouyer in ticket #1348):
	sys/arch/x86/x86/bus_dma.c: revision 1.54
	sys/arch/xen/x86/xen_bus_dma.c: revision 1.21
bus_dmamem_alloc() may not get a boundary smaller than size, but
it's perfectly valid for bus_dmamap_create() to do so (a contigous
transfers will then split in multiple segment).
Fix _xen_bus_dmamem_alloc_range() and _bus_dmamem_alloc_range() to
allow a boundary limit smaller than size:
- compute appropriate boundary for uvm_pglistalloc(), wich doesn't
  accept boundary < size
- also take care of boundary when deciding to start a new segment.
While there, remove useless boundary argument to _xen_alloc_contig().
Fix the boundary-related issue of PR port-amd64/42980

Sync with HEAD.

Sync with HEAD.

Machine dependent code is considered as part of UVM.  Include
internal API header.

Sync with HEAD

Sync with HEAD (-D20101022).

sync with head

fix copy/paste/modify error in a diagnostic panic message

sync with head.

sync with head

Sync with HEAD.

Adjustment for uvm/uvm_page.h.  More to follow later.

bus_dmamem_alloc() may not get a boundary smaller than size, but
it's perfectly valid for bus_dmamap_create() to do so (a contigous
transfers will then split in multiple segment).
Fix _xen_bus_dmamem_alloc_range() and _bus_dmamem_alloc_range() to
allow a boundary limit smaller than size:
- compute appropriate boundary for uvm_pglistalloc(), wich doesn't
  accept boundary < size
- also take care of boundary when deciding to start a new segment.
While there, remove useless boundary argument to _xen_alloc_contig().
Fix the boundary-related issue of PR port-amd64/42980

sync with head

Fixes regarding paddr_t/pd_entry_t types in MD x86 code, exposed by PAE:

- NBPD_* macros are set to the types that better match their architecture
(UL for i386 and amd64, ULL for i386 PAE) - will revisit when paddr_t is
set to 64 bits for i386 non-PAE.

- type fixes in printf/printk messages (Use PRIxPADDR when printing paddr_t
values, instead of %lx - paddr_t/pd_entry_t being 64 bits with PAE)

- remove casts that are no more needed now that Xen2 support has been dropped

Some fixes are from jmorse@ patches for PAE.

Compile + tested for i386 GENERIC and XEN3 kernels. Only compile tested for
amd64.

Reviewed by bouyer@.

See also http://mail-index.netbsd.org/tech-kern/2010/02/22/msg007373.html

Don't call _bus_dmamem_free() when _bus_dmamem_alloc() fails.
Fixes PR/42208

Sync with HEAD.

Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.

sync with head.

Sync with HEAD.

change pmap flags argument from int to u_int.
discussed with christos@ on source-changes-d@

Introduce PMAP_NOCACHE as first PMAP MD bit in x86. Make use of it in pmap_enter().
This safes one extra TLB flush when mapping dma-safe memory.
Presented on tech-kern@, port-i386@ and port-amd64@
ok ad@

Remove all the __P() from sys (excluding sys/dist)
Diff checked with grep and MK1 eyeball.
i386 and amd64 GENERIC and sys still build.

backout rev. 1.47.
per request from dyoung@ and cube@

bus_dmamap_create(): on failure, reset dmamp or drivers
like nfe(4) try to call bus_dmamap_destroy() on an invalid dmamap in their error path.

Sync with HEAD.

Sync with HEAD.

Update haad-dm branch to haad-dm-base2.

Typo in comment.

Pull up revisions:
  sys/arch/amd64/include/cpufunc.h	patch
  sys/arch/i386/include/cpufunc.h	patch
  sys/arch/x86/x86/bus_dma.c		1.45 via patch
requested by bouyer in ticket 1945.

Sync with wrstuden-revivesa-base-2.

Sync with netbsd-4.

Pull up revision 1.45 (requested by bouyer in ticket #1165).

port-i386/38935: Add appropriate x86_lfence or x86_mfence calls to
bus_dmamap_sync(), depending on the BUS_DMASYNC flag. This makes sure
that the kernel reads data from main memory in the intended order.

Sync with head.

Sync with HEAD.

port-i386/38935: Add appropriate x86_lfence or x86_mfence calls to
bus_dmamap_sync(), depending on the BUS_DMASYNC flag. This makes sure
that the kernel reads data from main memory in the intended order.

Sync w/ -current. 34 merge conflicts to follow.

Sync with head.

sync with head.

"functin" -> "function" (no "functional" change, har har)

Sync with HEAD.

Also fix build.

vm_page: put TAILQ_ENTRY into a union with LIST_ENTRY, so we can use both.

Sync with HEAD.

sync with head.

sync with head.

Remove clause 3 and 4 from TNF licenses

- Rename crit_enter/crit_exit to kpreempt_disable/kpreempt_enable.
  DragonflyBSD uses the crit names for something quite different.
- Add a kpreempt_disabled function for diagnostic assertions.
- Add inline versions of kpreempt_enable/kpreempt_disable for primitives.
- Make some more changes for preemption safety to the x86 pmap.

sync with HEAD

Sync with HEAD.

sync with head

Sync with HEAD.

Sync with HEAD.

Remove remaining CPUCLASS_386 tests.

sync with HEAD

sync with head.

Sync with HEAD.

Follow the merge of pmap.c on i386 and amd64 and move
pmap_init_tmp_pgtbl into arch/x86/x86/pmap.c. Modify the ACPI wakeup
code to restore CR4 before jumping back into kernel space as the large
page option might cover that.

amd64 (aka x86-64) support for Xen. Based on the OpenBSD port done by
Mathieu Ropert in 2006.
DomU-only for now. An INSTALL_XEN3_DOMU kernel with a ramdisk will boot to
sysinst if you're lucky. Often it panics because a runable LWP has
a NULL stack (really, it's all of l->l_addr which is has been zeroed out
while the process was on the queue !)
TODO:
- bug fixes :)
- Most of the xpq_* functions should be shared with xen/i386
- The xen/i386 assembly bootstrap code should be remplaced with the C
  version in xenamd64/amd64/xpmap.c
- see if a config(5) trick could allow to merge xenamd64 back to xen or amd64.

Merge the ppcoea-renovation branch to HEAD.

This branch was a major cleanup and rototill of many of the various OEA
cpu based PPC ports that focused on sharing as much code as possible
between the various ports to eliminate near-identical copies of files in
every tree.  Additionally there is a new PIC system that unifies the
interface to interrupt code for all different OEA ppc arches.  The work
for this branch was done by a variety of people, too long to list here.

TODO:
bebox still needs work to complete the transition to -renovation.
ofppc still needs a bunch of work, which I will be looking at.
ev64260 still needs to be renovated
amigappc was not attempted.

NOTES:
pmppc was removed as an arch, and moved to a evbppc target.

Sync with HEAD

sync with head.

crit_enter/crit_exit are now available.

Sync with head.

Sync with head.

sync with head.

Sync with HEAD

Sync with HEAD.

x86 changes for pcc and LKMs.

- Replace most inline assembly with proper functions. As a side effect
  this reduces the size of amd64 GENERIC by about 120kB, and i386 by a
  smaller amount. Nearly all of the inlines did something slow, or something
  that does not need to be fast.
- Make curcpu() and curlwp functions proper, unless __GNUC__ && _KERNEL.
  In that case make them inlines. Makes curlwp LKM and preemption safe.
- Make bus_space and bus_dma more LKM friendly.
- Share a few more files between the ports.
- Other minor changes.

Sync with HEAD.

sync with head.

Sync with HEAD.

Merge most x86 changes from the vmlocking branch, except the threaded soft
interrupt stuff. This is mostly comprised of changes to the pmap modules to
work on multiprocessor systems without kernel_lock, and changes to speed up
tlb shootdowns.

- Use pmap_pte_setbits, pmap_pte_clearbits.
- Shoot down a range instead of single pages in bus_space.c.

amd64 changes, as yet untested:

- Adapt to vmlocking branch.
- Apply TLB shootdown and pv allocation changes to the pmap.
- Make it build.

- Update PTEs atomically. Only shoot down if the mappings were in use.
- _bus_dmamem_unmap: don't shootdown, pmap_remove() will do it for us.
- Assume that pmap_update() will wait for the TLB shootdown to complete.

- When zeroing/copying pages, use SSE2 movtni to avoid polluting the cache.
- By default, align assembly routines on 32-byte starting boundaries.
- There are now 8 interrupt priority levels, half of which are softints.
  Update intrdefs.h to match.
- Always clear/set spinlock words - removes lots of ifdefs.
- Remove the horrible ci_self150 hack that I introduced.
- Overhaul how TLB shootdown is performed. Inspired by a similar change in
  OpenBSD but implemented quite differently. This should be a lot faster
  but I have not benchmarked it yet.

Sync with HEAD.

Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

- sync with head.
- move sched_changepri back to kern_synch.c as it doesn't know PPQ anymore.

sync with head.

add a pair of new bus_dma(9) functions:
	int _bus_dmatag_subregion(bus_dma_tag_t tag,
				  bus_addr_t min_addr,
				  bus_addr_t max_addr,
				  bus_dma_tag_t *newtag,
				  int flags)
	void _bus_dmatag_destroy(bus_dma_tag_t tag)

that allow a (normally broken/limited) device to restrict the bus address
range it can talk to.  this is used by bce(4) to limit DMA addresses to
1GB range, the maximum the chip can address.

all this is from Yorick Hardy <yhardy@uj.ac.za> with input from several
people on tech-kern.

XXX: bus_dma(9) needs an update still.

Merge newlock2 to head.

Remove support for SA. Ok core@.

sync with head.

sync with head.

Sync with head.

__unused removal on arguments; approved by core.

sync with head

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

Pull up following revision(s) (requested by bouyer in ticket #1510):
	sys/arch/xen/x86/xen_bus_dma.c: revision 1.7
	sys/arch/xen/x86/xen_bus_dma.c: revision 1.8
	sys/arch/x86/include/bus_private.h: revision 1.6
	sys/arch/x86/x86/bus_dma.c: revision 1.30
	sys/arch/xen/include/bus_private.h: revision 1.7
Some bus_dma(9) fixes for Xen:
- Attempt to gracefully recover from a failed decrease_reservation or
  increase_reservation, by avoiding physical memory loss.
- always store a machine address in ds_addr; this avoids some mistakes
  where machine address would in some case be freed at physical address, or
  mapped as physical address.
Wrap some printfs in #ifdef DEBUG, as we should not leak memory any more when
bus_dma memory allocation fails.

Pull up following revision(s) (requested by bouyer in ticket #150):
	sys/arch/xen/x86/xen_bus_dma.c: revision 1.7
	sys/arch/xen/x86/xen_bus_dma.c: revision 1.8
	sys/arch/x86/include/bus_private.h: revision 1.6
	sys/arch/x86/x86/bus_dma.c: revision 1.30
	sys/arch/xen/include/bus_private.h: revision 1.7
Some bus_dma(9) fixes for Xen:
- Attempt to gracefully recover from a failed decrease_reservation or
  increase_reservation, by avoiding physical memory loss.
- always store a machine address in ds_addr; this avoids some mistakes
  where machine address would in some case be freed at physical address, or
  mapped as physical address.
Wrap some printfs in #ifdef DEBUG, as we should not leak memory any more when
bus_dma memory allocation fails.

sync with head

sync with head.

Some bus_dma(9) fixes for Xen:
- Attempt to gracefully recover from a failed decrease_reservation or
  increase_reservation, by avoiding physical memory loss.
- always store a machine address in ds_addr; this avoids some mistakes
  where machine address would in some case be freed at physical address, or
  mapped as physical address.

sync with head.

Sync with head.

merge yamt-uio_vmspace branch.

- use vmspace rather than proc or lwp where appropriate.
  the latter is more natural to specify an address space.
  (and less likely to be abused for random purposes.)
- fix a swdmover race.

adapt x86 bus_dma.

sync with head.

Protect against uio_lwp being NULL from Pavel Cahyna

Remove leading __ from __(const|inline|signed|volatile) -- it is obsolete.

merge ktrace-lwp.

Sync with head.

sync with head.

bus_dmamem_map: honour BUS_DMA_NOWAIT.  noted by Manuel Bouyer.
bus_space_map: always do NOWAIT allocation as it used to be before yamt-km.

we have too many copies!

Sync with HEAD. Here we go again...

Turn bounce buffer stats into evcnts and enable them by default.

Pull up following revision(s) (requested by bouyer in ticket #697):
	sys/arch/x86/x86/bus_dma.c: revision 1.23
	sys/arch/x86/include/bus_private.h: revision 1.3
	sys/arch/xen/include/bus_private.h: revision 1.3
Rename _PRIVATE_BUS_DMAMEM_ALLOC_RANGE to _BUS_DMAMEM_ALLOC_RANGE for
consistency with other macros defined in bus_private.h. Pointed out by
YAMAMOTO Takashi.

Pull up following revision(s) (requested by bouyer in ticket #695):
	sys/arch/x86/x86/bus_dma.c: revision 1.22
	sys/arch/x86/include/bus_private.h: revision 1.2
More adjustements to deal with Xen's physical <=> machine addresses mappings:
- Allow _bus_dmamem_alloc_range to be provided from external source:
  Use a _PRIVATE_BUS_DMAMEM_ALLOC_RANGE macro, defined to
  _bus_dmamem_alloc_range by default.
- avail_end is the end of the physical address range. Define a macro
 _BUS_AVAIL_END (defined by default to avail_end) and use it instead.

Rename _PRIVATE_BUS_DMAMEM_ALLOC_RANGE to _BUS_DMAMEM_ALLOC_RANGE for
consistency with other macros defined in bus_private.h. Pointed out by
YAMAMOTO Takashi.

More adjustements to deal with Xen's physical <=> machine addresses mappings:
- Allow _bus_dmamem_alloc_range to be provided from external source:
  Use a _PRIVATE_BUS_DMAMEM_ALLOC_RANGE macro, defined to
  _bus_dmamem_alloc_range by default.
- avail_end is the end of the physical address range. Define a macro
 _BUS_AVAIL_END (defined by default to avail_end) and use it instead.

sync with -current

Pull up revision 1.21 (requested by yamt in ticket #175):
tweak x86 bus_dma code so that it can be used by xen port.
- distinguish paddr_t and bus_addr_t.
  for xen, use bus_addr_t in the sense of machine address.
- move _X86_BUS_DMA_PRIVATE part of bus.h into bus_private.h.
- remove special handling of xen_shm.  we can always grab
  machine address from pte.

tweak x86 bus_dma code so that it can be used by xen port.

- distinguish paddr_t and bus_addr_t.
  for xen, use bus_addr_t in the sense of machine address.
- move _X86_BUS_DMA_PRIVATE part of bus.h into bus_private.h.
- remove special handling of xen_shm.  we can always grab
  machine address from pte.

Sync with HEAD.

merge yamt-km branch.
- don't use managed mappings/backing objects for wired memory allocations.
  save some resources like pv_entry.  also fix (most of) PR/27030.
- simplify kernel memory management API.
- simplify pmap bootstrap of some ports.
- some related cleanups.

sync with head.  xen and whitespace.  xen part is not finished.

Add a dm_maxsegsz public member to bus_dmamap_t.  This allows a user of the API
to select the maximum segment size for each bus_dmamap_load (up to the maxsegsz
supplied to bus_dmamap_create).  dm_maxsegsz is reset to the value supplied to
bus_dmamap_create when the dmamap is unloaded.

Sync with HEAD.

Hi Perry!

g/c obsolete comment for _bus_dmamap_load_buffer()

- convert i386 to new apis.
- remove a pmap bootstrap kludge, which is no longer needed.

Fix the sync with head I botched.

Sync with HEAD.

Sync with HEAD

Remove the "ID" component of the x86 bus_dma flags, since these are no
longer "ISA DMA" specific flags.

remove XXX comments which are no longer true.

ANSIfy.

simplify x86 bus_dma implementation.
(rather than passing &lastaddr and &seg around,
use and update bus_dmamap_t directly.)

- introduce _bus_dmamap_load_paddr, which takes (paddr, size) and
  add the range to the map, and use it for _bus_dmamap_load_{buffer,mbuf}.
- _bus_dmamap_load_mbuf: in the case of M_EXT_PAGES, deal with vm_pages
  directly rather than doing pmap_extract on given kva.

as a side effect, do a segment coalescing and boundary checks for mbufs.

ok'ed by Frank van der Linden and Jason Thorpe on tech-kern@.

simplify x86 bus_dma internal "load" functions.
(by eliminating a variable "first" and using seg == -1 instead.)

unexport following x86 bus_dma internal functions.
        _bus_dma_alloc_bouncebuf
        _bus_dma_free_bouncebuf
        _bus_dmamap_load_buffer

_bus_dmamap_load_mbuf: check bounce_thresh in the case when we have paddr hint.

In _bus_dma_uiomove():
* Don't punt after the first iov in the UIO_SYSSPACE case.  Not that this ever
  happens in practice right now.
* If we get through the loop, error==0 by definition, so just return 0.
* Eliminate bogus initializer.

Fix uninitialized variable warning

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.

Apply the aborted ktrace-lwp changes to a specific branch.  This is just for
others to review, I'm concerned that patch fuziness may have resulted in some
errant code being generated but I'll look at that later by comparing the diff
from the base to the branch with the file I attempt to apply to it.  This will,
at the very least, put the changes in a better context for others to review
them and attempt to tinker with removing passing of 'struct lwp' through
the kernel.

Back out the lwp/ktrace changes. They contained a lot of colateral damage,
and need to be examined and discussed more.

Pass lwp pointers throughtout the kernel, as required, so that the lwpid can
be inserted into ktrace records.  The general change has been to replace
"struct proc *" with "struct lwp *" in various function prototypes, pass
the lwp through and use l_proc to get the process pointer when needed.

Bump the kernel rev up to 1.6V

Avoid bad free() calls for failed allocations. From enami.

Generalize bounce buffers, and use them for 32 bit PCI if needed.
Make ALLOCNOW the default iff bouncing might be needed (this has
no effect on i386 because ISA DMA devices already had to use
ALLOCNOW, and PCI isn't bounced (yet), since we don't do > 4G
at this point for i386.

Use cached physical addresses for mbufs and clusters to save having
to extract the physical address from the virtual.

On the ARM, also use the "read-only at MMU" indication to avoid a
redundant cache clean operation.

Other platforms should use these two as examples of how to use these
new pool/mbuf features to improve network performance.  Note this requires
a platform to provide a working POOL_VTOPHYS().

Part 3 in a series of simple patches contributed by Wasabi Systems
to improve network performance.

Split bus_space and bus_dma into separate files.