CVS log for src/sys/uvm/uvm_page.c

Up to [cvs.NetBSD.org] / src / sys / uvm

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

Rename the local "boot_cpu" variable to "uvm_boot_cpu".

s/musn't/mustn't/ in comments.

uvm_phys_to_vm_page() turns out to be a fairly central routine due to the
way that some of the pmaps work, so try to optimise it a little.

uvm(9): One rndsource for faults -- not one per CPU.

All relevant state is per-CPU anyway; the only substantive difference
this makes is how many entries appear in `rndctl -l' output and what
they are called -- formerly the somewhat confusing `cpuN', meaning
`page faults on cpuN', and now just `uvmfault'.  I don't think
there's any real value in being able to enable or disable measurement
or counting of page faults on one CPU vs others, so although this
could be a minor compatibility change, it's hard to imagine it
matters much.

XXX kernel ABI change in struct cpu_info

uvm(9): KASSERT(A && B) -> KASSERT(A); KASSERT(B)

ddb/db_active.h: New home for extern db_active.

This can be included unconditionally, and db_active can then be
queried unconditionally; if DDB is not in the kernel, then db_active
is a constant zero.  Reduces need for #include opt_ddb.h, #ifdef DDB.

Pull up following revision(s) - all via patch -
(requested by riastradh in ticket #1317):

	sys/uvm/uvm_page.c: revision 1.248
	sys/uvm/uvm_anon.c: revision 1.80
	sys/rump/librump/rumpvfs/vm_vfs.c: revision 1.40
	sys/rump/librump/rumpvfs/vm_vfs.c: revision 1.41
	sys/rump/librump/rumpkern/vm.c: revision 1.191
	sys/uvm/uvm_pager.c: revision 1.130
	external/cddl/osnet/dist/uts/common/fs/zfs/zfs_vnops.c: revision 1.71
	tests/rump/rumpkern/t_vm.c: revision 1.5
	tests/rump/rumpkern/t_vm.c: revision 1.6
	sys/rump/librump/rumpvfs/vm_vfs.c: revision 1.39

Move the handling of PG_PAGEOUT from uvm_aio_aiodone_pages() to
uvm_page_unbusy() so that all callers of uvm_page_unbusy() don't need to
handle this flag separately.  Split out the pages part of uvm_aio_aiodone()
into uvm_aio_aiodone_pages() in rump just like in the real kernel.

In ZFS functions that can fail to copy data between the ARC and VM pages,
use uvm_aio_aiodone_pages() rather than uvm_page_unbusy() so that we can
handle these "I/O" errors.  Fixes PR 55702.

fix an incorrect assertion in the previous commit.

Handle PG_PAGEOUT in uvm_anon_release() too.

Commit the ZFS file that I forgot in this previous commit:

Move the handling of PG_PAGEOUT from uvm_aio_aiodone_pages() to
uvm_page_unbusy() so that all callers of uvm_page_unbusy() don't need to
handle this flag separately.  Split out the pages part of uvm_aio_aiodone()
into uvm_aio_aiodone_pages() in rump just like in the real kernel.

In ZFS functions that can fail to copy data between the ARC and VM pages,
use uvm_aio_aiodone_pages() rather than uvm_page_unbusy() so that we can
handle these "I/O" errors.  Fixes PR 55702.
update the rump copy of uvm_page_unbusy() to match the real version,
in particular handle PG_PAGEOUT.  fixes a few atf tests.
the busypage test is buggy, expect it to fail.

make rump's uvm_aio_aiodone_pages() look more like the kernel version.
fixes some more rumpy assertions.

for the busypage test, replace atf_tc_expect_fail() with atf_tc_skip()
because atf apparently has no way to expect a test program to crash.
fixes PR 55945.

Sync w/ HEAD.

Some KNF.  NFC.

In the current code, CPU_COUNT_FREEPAGES counts pages in the global
freelists AND the per-CPU pgflcache free pages caches, and that is the
number of pages that the pagedaemon considers to be available.
However, most pages in the pgflcache per-CPU free page caches are NOT
actually available for any particular allocation, and thus allocating
a page can fail even though the pagedaemon thinks enough pages are
available.  This change makes CPU_COUNT_FREEPAGES only count pages in
the global freelists and not pages in the pgflcache per-CPU free page
caches, thus better aligning the pagedaemon's view of how many pages
are available with the number of pages that can actually be allocated
by any particular request.  This fixes a hang that Christos was hitting.

Move the handling of PG_PAGEOUT from uvm_aio_aiodone_pages() to
uvm_page_unbusy() so that all callers of uvm_page_unbusy() don't need to
handle this flag separately.  Split out the pages part of uvm_aio_aiodone()
into uvm_aio_aiodone_pages() in rump just like in the real kernel.
In ZFS functions that can fail to copy data between the ARC and VM pages,
use uvm_aio_aiodone_pages() rather than uvm_page_unbusy() so that we can
handle these "I/O" errors.  Fixes PR 55702.

G/C uvm_pagezerocheck

add a __diagused to fix non-DIAGNOSTIC kernel

centralize calls from UVM to radixtree into a few functions.
in those functions, assert that the object lock is held in
the correct mode.

Consistently use UVMHIST(__func__)

Convert UVMHIST_{CALLED,LOG} into UVMHIST_CALLARGS

<sys/extent.h> not needed here.

Remove PG_ZERO.  It worked brilliantly on x86 machines from the mid-90s but
having spent an age experimenting with it over the last 6 months on various
machines and with different use cases it's always either break-even or a
slight net loss for me.

uvm_pagerealloc(): resurrect the insertion case.

Counter tweaks:

- Don't need to count anonpages+filepages any more; clean+unknown+dirty for
  each kind of page can be summed to get the totals.

- Track the number of free pages with a counter so that it's one less thing
  for the allocator to do, which opens up further options there.

- Remove cpu_count_sync_one().  It has no users and doesn't save a whole lot.
  For the cheap option, give cpu_count_sync() a boolean parameter indicating
  that a cached value is okay, and rate limit the updates for cached values
  to hz.

uvm_availmem(): give it a boolean argument to specify whether a recent
cached value will do, or if the very latest total must be fetched.  It can
be called thousands of times a second and fetching the totals impacts not
only the calling LWP but other CPUs doing unrelated activity in the VM
system.

Add uvm_pagewanted_p(): return true if someone is waiting on the page and
assert caller has correct lock to observe that.

UVM_PAGE_TRKOWN: print the LID too

Don't set PG_AOBJ on a page unless UVM_OBJ_IS_AOBJ(), otherwise it can
catch pages from e.g. uvm_loanzero_object.

- If the hardware provided NUMA info, then use it to decide how to set up
  the allocator's buckets, instead of doing round robin distribution.  There
  are open questions here but this is better than doing nothing.

- Kernel reserve pages are for the kernel not realtime threads.

Mostly merge changes from HEAD upto 20200411

Merge changes from current as of 20200406

Tweak the March 14th change to make page waits interlocked by pg->interlock.
Remove unneeded changes and only deal with the PQ_WANTED flag, to exclude
possible bugs.

Fix build with UVMHIST.

Don't require a write lock for page enqueue/activate/deactivate.

Make page waits (WANTED vs BUSY) interlocked by pg->interlock.  Gets RW
locks out of the equation for sleep/wakeup, and allows observing+waiting
for busy pages when holding only a read lock.  Proposed on tech-kern.

Trailing whitespace

Typo in comment

Sync with head.

Tighten up the locking around vp->v_iflag a little more after the recent
split of vmobjlock & v_interlock.

Fix a comment.

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart.  v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap.  Others to follow later.

Sync with head.

uvmpdpol_pageactive(): the change to not re-activate recently activated
pages worked great with uvm_pageqlock, but it doesn't buy anything any more,
because now the busy pages are likely in a per-CPU queue somewhere waiting
to be processed, and changing the intent on those queued pages costs next
to nothing.  Remove this and get back all the bits in pg->pqflags.

Sync with head.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
  pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
  precisely in uvm layer.

- uvm_pagezerocheck(): put a global lock around it to protect the single
  page mapping (DEBUG only).

- uvm_pagefree(): increment zeropages as needed.

- Many small tweaks to the SMT awareness in the scheduler.  It does a much
  better job now at keeping all physical CPUs busy, while using the extra
  threads to help out.  In particular, during preempt() if we're using SMT,
  try to find a better CPU to run on and teleport curlwp there.

- Change the CPU topology stuff so it can work on asymmetric systems.  This
  mainly entails rearranging one of the CPU lists so it makes sense in all
  configurations.

- Add a parameter to cpu_topology_set() to note that a CPU is "slow", for
  where there are fast CPUs and slow CPUs, like with the Rockwell RK3399.
  Extend the SMT awareness to try and handle that situation too (keep fast
  CPUs busy, use slow CPUs as helpers).

Page allocator:

The method for assigning pages to buckets in the non-NUMA case sucks.  It
can defeat memory interleaving in the hardware, and not distribute pages
fairly by colour.  To fix this and make things more deterministic, take the
physical PFN and colour into account.

Then when freeing pages, in the non-NUMA case don't change the page's bucket
either.  Keeping the bucket number stable will also permit partitioning page
replacement state by CPU package / NUMA node.

- Add and use wrapper functions that take and acquire page interlocks, and pairs
  of page interlocks.  Require that the page interlock be held over calls to
  uvm_pageactivate(), uvm_pagewire() and similar.

- Solve the concurrency problem with page replacement state.  Rather than
  updating the global state synchronously, set an intended state on
  individual pages (active, inactive, enqueued, dequeued) while holding the
  page interlock.  After the interlock is released put the pages on a 128
  entry per-CPU queue for their state changes to be made real in batch.
  This results in in a ~400 fold decrease in contention on my test system.
  Proposed on tech-kern but modified to use the page interlock rather than
  atomics to synchronise as it's much easier to maintain that way, and
  cheaper.

Rename uvm_free() -> uvm_availmem().

Rename uvm_page_locked_p() -> uvm_page_owner_locked_p()

uvm_pagealloc_pgb(): don't fill cache if we're into the reserves.

uvm_pagereplace(): use radix_tree_replace_node() to avoid alloc/free.

Add missing call to uvm_pgflcache_resume().

Use PRIxPADDR to print a physical address (instead of casting to void*
and printing a pointer - which does not work well if sizeof(paddr_t) !=
sizeof(void*)).

Nothing uses uvm.cpus any more, and we can do the same with cpu_lookup(),
so get rid of it.

Redo the page allocator to perform better, especially on multi-core and
multi-socket systems.  Proposed on tech-kern.  While here:

- add rudimentary NUMA support - needs more work.
- remove now unused "listq" from vm_page.

uvm_pagealloc_strat(): Tweak the locking to allow for lazy dequeue of pages
in the pdpolicy code.  This means taking pg->interlock if assigning to
an object.  The remaining barrier to lazy dequeue is having a dedicated
TAILQ_ENTRY in the page (it's currently shared with the page allocator).

uvm_page_to_phys: mask off the lower bits.

Detangle the pagedaemon from uvm_fpageqlock:

- Have a single lock (uvmpd_lock) to protect pagedaemon state that was
  previously covered by uvmpd_pool_drain_lock plus uvm_fpageqlock.
- Don't require any locks be held when calling uvm_kick_pdaemon().
- Use uvm_free().

- Rename VM_PGCOLOR_BUCKET() to VM_PGCOLOR().  I want to reuse "bucket" for
  something else soon and TBH it matches what this macro does better.

- Add inlines to set/get locator values in the unused lower bits of
  pg->phys_addr.  Begin by using it to cache the freelist index, because
  computing it is expensive and that shows up during profiling.  Discussed
  on tech-kern.

Counter tweaks:

"zeroaborts" + "free" don't need to be per-CPU counters, and "bucketmiss"
wasn't used.  Remove those and cluster by usage.

Add uvm_free(): returns number of free pages in system.

PR kern/54783: t_mmap crahes the kernel

- Fix various locking & sequencing errors with breaking loans.

- Don't call uvm_pageremove_tree() while holding pg->interlock as radixtree
  can take further locks when freeing nodes.

- Extend the per-CPU counters matt@ did to include all of the hot counters
  in UVM, excluding uvmexp.free, which needs special treatment and will be
  done with a separate commit.  Cuts system time for a build by 20-25% on
  a 48 CPU machine w/DIAGNOSTIC.

- Avoid 64-bit integer divide on every fault (for rnd_add_uint32).

Merge from yamt-pagecache:

uvm_pagerealloc(): Don't bother with insert to new.  Nobody uses it and it
can return an error now due to radixtree.

Merge from yamt-pagecache:

- do gang lookup of pages using radixtree.
- remove now unused uvm_object::uo_memq and vm_page::listq.queue.

Merge from yamt-pagecache: use radixtree for page lookup.

rbtree page lookup was introduced during the NetBSD 5.0 development cycle to
bypass lock contention problems with the (then) global page hash, and was a
temporary solution to allow us to make progress.  radixtree is the intended
replacement.

Ok yamt@.

Break the global uvm_pageqlock into a per-page identity lock and a private
lock for use of the pagedaemon policy code.  Discussed on tech-kern.

PR kern/54209: NetBSD 8 large memory performance extremely low
PR kern/54210: NetBSD-8 processes presumably not exiting
PR kern/54727: writing a large file causes unreasonable system behaviour

Fix programming mistake: 'paddrp' is a pointer given as argument, setting
it to NULL in the called function does not set it to NULL in the caller.

Actually, the callers of these functions do not do anything with the
special error handling, so drop the unused checks and the NULL assignments
altogether.

Found by the lgtm bot.

Sync with HEAD

Avoid a panic from the sequence

	mlock(buf, 0);
	munlock(buf, 0);
	mlock(buf, page);
	munlock(buf, page);

where buf is page aligned, and page is actually anything > 0
(but not too big) which will get rounded up to the next multiple
of the page size.

In that sequence, it is possible that the 1st munlock() is optional.

Add a KASSERT() (or two) to detect the first effects of the problem
(without that, or in !DIAGNOSTIC kernels) the problem eventually
causes some kind of problem or other (most often still a panic.)

After this, mlock(anything, 0) (or munlock) validates "anything"
but is otherwise a no-op (regardless of the alignment of anything).

Also, don't treat mlock(buf, verybig) as equivalent to mlock(buf, 0)
which is (more or less) what we had been doing.

XXX pullup -8 (maybe -7 as well, need to check).

Sync with HEAD

add experimental new function uvm_direct_process(), to allow of read/writes
of contents of uvm pages without mapping them into kernel, using
direct map or moral equivalent; pmaps supporting the interface need
to provide pmap_direct_process() and define PMAP_DIRECT

implement the new interface for amd64; I hear alpha and mips might be relatively
easy to add too, but I lack the knowledge

part of resolution for PR kern/53124

detect wraparound when bumping page wire_count and loan_count

Synch with HEAD

In uvm_page_recolor(), kmem_free() old size rather than new size.

From Yaniv Abraham-Rabinovitch in PR kern/53208.

update from HEAD

add two new members to uvmexp_sysctl{}:  bootpages and poolpages.
bootpages is set to the pages allocated via uvm_pageboot_alloc().
poolpages is calculated from the list of pools nr_pages members.

this brings us closer to having a valid total of pages known by
the system, vs actual pages originally managed.

XXX: poolpages needs some handling for PR_RECURSIVE pools still.

Pull up following revision(s) (requested by pgoyette in ticket #335):
	share/man/man9/kernhist.9: 1.5-1.8
	sys/arch/acorn26/acorn26/pmap.c: 1.39
	sys/arch/arm/arm32/fault.c: 1.105 via patch
	sys/arch/arm/arm32/pmap.c: 1.350, 1.359
	sys/arch/arm/broadcom/bcm2835_bsc.c: 1.7
	sys/arch/arm/omap/if_cpsw.c: 1.20
	sys/arch/arm/omap/tiotg.c: 1.7
	sys/arch/evbarm/conf/RPI2_INSTALL: 1.3
	sys/dev/ic/sl811hs.c: 1.98
	sys/dev/usb/ehci.c: 1.256
	sys/dev/usb/if_axe.c: 1.83
	sys/dev/usb/motg.c: 1.18
	sys/dev/usb/ohci.c: 1.274
	sys/dev/usb/ucom.c: 1.119
	sys/dev/usb/uhci.c: 1.277
	sys/dev/usb/uhub.c: 1.137
	sys/dev/usb/umass.c: 1.160-1.162
	sys/dev/usb/umass_quirks.c: 1.100
	sys/dev/usb/umass_scsipi.c: 1.55
	sys/dev/usb/usb.c: 1.168
	sys/dev/usb/usb_mem.c: 1.70
	sys/dev/usb/usb_subr.c: 1.221
	sys/dev/usb/usbdi.c: 1.175
	sys/dev/usb/usbdi_util.c: 1.67-1.70
	sys/dev/usb/usbroothub.c: 1.3
	sys/dev/usb/xhci.c: 1.75
	sys/external/bsd/drm2/dist/drm/i915/i915_gem.c: 1.34
	sys/kern/kern_history.c: 1.15
	sys/kern/kern_xxx.c: 1.74
	sys/kern/vfs_bio.c: 1.275-1.276
	sys/miscfs/genfs/genfs_io.c: 1.71
	sys/sys/kernhist.h: 1.21
	sys/ufs/ffs/ffs_balloc.c: 1.63
	sys/ufs/lfs/lfs_vfsops.c: 1.361
	sys/ufs/lfs/ulfs_inode.c: 1.21
	sys/ufs/lfs/ulfs_vnops.c: 1.52
	sys/ufs/ufs/ufs_inode.c: 1.102
	sys/ufs/ufs/ufs_vnops.c: 1.239
	sys/uvm/pmap/pmap.c: 1.37-1.39
	sys/uvm/pmap/pmap_tlb.c: 1.22
	sys/uvm/uvm_amap.c: 1.108
	sys/uvm/uvm_anon.c: 1.64
	sys/uvm/uvm_aobj.c: 1.126
	sys/uvm/uvm_bio.c: 1.91
	sys/uvm/uvm_device.c: 1.66
	sys/uvm/uvm_fault.c: 1.201
	sys/uvm/uvm_km.c: 1.144
	sys/uvm/uvm_loan.c: 1.85
	sys/uvm/uvm_map.c: 1.353
	sys/uvm/uvm_page.c: 1.194
	sys/uvm/uvm_pager.c: 1.111
	sys/uvm/uvm_pdaemon.c: 1.109
	sys/uvm/uvm_swap.c: 1.175
	sys/uvm/uvm_vnode.c: 1.103
	usr.bin/vmstat/vmstat.c: 1.219
Reorder to test for null before null deref in debug code
--
Reorder to test for null before null deref in debug code
--
KNF
--
No need for '\n' in UVMHIST_LOG
--
normalise a BIOHIST log message
--
Update the kernhist(9) kernel history code to address issues identified
in PR kern/52639, as well as some general cleaning-up...
(As proposed on tech-kern@ with additional changes and enhancements.)
Details of changes:
* All history arguments are now stored as uintmax_t values[1], both in
   the kernel and in the structures used for exporting the history data
   to userland via sysctl(9).  This avoids problems on some architectures
   where passing a 64-bit (or larger) value to printf(3) can cause it to
   process the value as multiple arguments.  (This can be particularly
   problematic when printf()'s format string is not a literal, since in
   that case the compiler cannot know how large each argument should be.)
* Update the data structures used for exporting kernel history data to
   include a version number as well as the length of history arguments.
* All [2] existing users of kernhist(9) have had their format strings
   updated.  Each format specifier now includes an explicit length
   modifier 'j' to refer to numeric values of the size of uintmax_t.
* All [2] existing users of kernhist(9) have had their format strings
   updated to replace uses of "%p" with "%#jx", and the pointer
   arguments are now cast to (uintptr_t) before being subsequently cast
   to (uintmax_t).  This is needed to avoid compiler warnings about
   casting "pointer to integer of a different size."
* All [2] existing users of kernhist(9) have had instances of "%s" or
   "%c" format strings replaced with numeric formats; several instances
   of mis-match between format string and argument list have been fixed.
* vmstat(1) has been modified to handle the new size of arguments in the
   history data as exported by sysctl(9).
* vmstat(1) now provides a warning message if the history requested with
   the -u option does not exist (previously, this condition was silently
   ignored, with only a single blank line being printed).
* vmstat(1) now checks the version and argument length included in the
   data exported via sysctl(9) and exits if they do not match the values
   with which vmstat was built.
* The kernhist(9) man-page has been updated to note the additional
   requirements imposed on the format strings, along with several other
   minor changes and enhancements.
[1] It would have been possible to use an explicit length (for example,
     uint64_t) for the history arguments.  But that would require another
     "rototill" of all the users in the future when we add support for an
     architecture that supports a larger size.  Also, the printf(3)
format
     specifiers for explicitly-sized values, such as "%"PRIu64, are much
     more verbose (and less aesthetically appealing, IMHO) than simply
     using "%ju".
[2] I've tried very hard to find "all [the] existing users of
kernhist(9)"
     but it is possible that I've missed some of them.  I would be glad
to
     update any stragglers that anyone identifies.
--
For some reason this single kernel seems to have outgrown its declared
size as a result of the kernhist(9) changes.  Bump the size.
XXX The amount of increase may be excessive - anyone with more detailed
XXX knowledge please feel free to further adjust the value
appropriately.
--
Misssed one cast of pointer --> uintptr_t in previous kernhist(9) commit
--
And yet another one.  :(
--
Use correct mark-up for NetBSD version.
--
More improvements in grammar and readability.
--
Remove a stray '"' (obvious typo) and add a couple of casts that are
probably needed.
--
And replace an instance of "%p" conversion with "%#jx"
--
Whitespace fix. Give Bl tag table a width. Fix Xr.

Update the kernhist(9) kernel history code to address issues identified
in PR kern/52639, as well as some general cleaning-up...

(As proposed on tech-kern@ with additional changes and enhancements.)

Details of changes:

* All history arguments are now stored as uintmax_t values[1], both in
  the kernel and in the structures used for exporting the history data
  to userland via sysctl(9).  This avoids problems on some architectures
  where passing a 64-bit (or larger) value to printf(3) can cause it to
  process the value as multiple arguments.  (This can be particularly
  problematic when printf()'s format string is not a literal, since in
  that case the compiler cannot know how large each argument should be.)

* Update the data structures used for exporting kernel history data to
  include a version number as well as the length of history arguments.

* All [2] existing users of kernhist(9) have had their format strings
  updated.  Each format specifier now includes an explicit length
  modifier 'j' to refer to numeric values of the size of uintmax_t.

* All [2] existing users of kernhist(9) have had their format strings
  updated to replace uses of "%p" with "%#jx", and the pointer
  arguments are now cast to (uintptr_t) before being subsequently cast
  to (uintmax_t).  This is needed to avoid compiler warnings about
  casting "pointer to integer of a different size."

* All [2] existing users of kernhist(9) have had instances of "%s" or
  "%c" format strings replaced with numeric formats; several instances
  of mis-match between format string and argument list have been fixed.

* vmstat(1) has been modified to handle the new size of arguments in the
  history data as exported by sysctl(9).

* vmstat(1) now provides a warning message if the history requested with
  the -u option does not exist (previously, this condition was silently
  ignored, with only a single blank line being printed).

* vmstat(1) now checks the version and argument length included in the
  data exported via sysctl(9) and exits if they do not match the values
  with which vmstat was built.

* The kernhist(9) man-page has been updated to note the additional
  requirements imposed on the format strings, along with several other
  minor changes and enhancements.

[1] It would have been possible to use an explicit length (for example,
    uint64_t) for the history arguments.  But that would require another
    "rototill" of all the users in the future when we add support for an
    architecture that supports a larger size.  Also, the printf(3) format
    specifiers for explicitly-sized values, such as "%"PRIu64, are much
    more verbose (and less aesthetically appealing, IMHO) than simply
    using "%ju".

[2] I've tried very hard to find "all [the] existing users of kernhist(9)"
    but it is possible that I've missed some of them.  I would be glad to
    update any stragglers that anyone identifies.

Sync with HEAD

remove checks for failure after memory allocation calls that cannot fail:

  kmem_alloc() with KM_SLEEP
  kmem_zalloc() with KM_SLEEP
  percpu_alloc()
  pserialize_create()
  psref_class_create()

all of these paths include an assertion that the allocation has not failed,
so callers should not assert that again.

Sync with HEAD

Sync with HEAD

Sync with HEAD

Fix off by one.
ok cherry

Sync with HEAD.  (Note that most of these changes are simply $NetBSD$
tag issues.)

Fix uvm_page_physget_freelist so that it actually performs the two passes
it mentions.

"Make NetBSD great again!"

Introduce uvm_hotplug(9) to the kernel.

Many thanks, in no particular order to:

TNF, for funding the project.

Chuck Silvers - for multiple API reviews and feedback.
Nick Hudson - for testing on multiple architectures and bugfix patches.
Everyone who helped with boot testing.

KeK (http://www.kek.org.in) for hosting the primary developers.

physmem should be of type psize_t

Also, use PRIxPSIZE when printf(9)ing physmem.

Use uvm_physseg.h:uvm_page_physload() instead of uvm_extern.h

For this, include uvm_physseg.h in the build and include tree, make a
cosmetic modification to the prototype for uvm_page_physload().

Sync with HEAD

Allow changing the per-cpu emergency page reservation via kernel config.

Don't use C++ try keyword as a variable name.

Rebase to HEAD as of a few days ago.

Merge tls-earlyentropy branch into HEAD.

Rebase.

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs.  ("Protocol error: too many arguments")

sync with head

remove unused variables for UVM_PAGE_TRKOWN.

Be a little more clear and consistent about harvesting entropy from devices:

1) deprecate RND_FLAG_NO_ESTIMATE

2) define RND_FLAG_COLLECT_TIME, RND_FLAG_COLLECT_VALUE

3) define RND_FLAG_ESTIMATE_TIME, RND_FLAG_ESTIMATE_VALUE

4) define RND_FLAG_DEFAULT: RND_FLAG_COLLECT_TIME|
   RND_FLAG_COLLECT_VALUE|RND_FLAG_ESTIMATE_TIME

5) Make entropy harvesting from environmental sensors a little more generic
   and remove it from individual sensor drivers.

6) Remove individual open-coded delta-estimators for values from a few
   places in the tree (uvm, environmental drivers).

7) 0 -> RND_FLAG_DEFAULT, actually gather entropy from various drivers
   that had stubbed out code, other minor cleanups.

Make sure the hint is initialized to NULL.

Adapt to K{,D}ASSERTMSG changes

Mark a diagnostic-only variable

fix an assertion

sync with head

Improve UVM_PAGE_TRKOWN.
Add more asserts to uvm_page.

merge to -current.

byebye PG_HOLE as it turned out to be unnecessary.

Add KASSERTs to uvm_pagealloc_pgfl to verify the page is actually free and has
the contents that it should.
Redo the KASSERTs for the pageq in uvm_pagefree.

Track the victims selected by the pagedaemon and what happens to then.
Keep a hint for what page group has the most free pages for a given color.

Add more KASSERTs (more! more! more!).
When returning page to the free pool, make sure to dequeue the pages before
hand or free page queue corruption will happen.

Major changes to uvm.
Support multiple collections (groups) of free pages and run the page
reclaimation algorithm on each group independently.

Entropy-pool implementation move and cleanup.

1) Move core entropy-pool code and source/sink/sample management code
   to sys/kern from sys/dev.

2) Remove use of NRND as test for presence of entropy-pool code throughout
   source tree.

3) Remove use of RND_ENABLED in device drivers as microoptimization to
   avoid expensive operations on disabled entropy sources; make the
   rnd_add calls do this directly so all callers benefit.

4) Fix bug in recent rnd_add_data()/rnd_add_uint32() changes that might
   have lead to slight entropy overestimation for some sources.

5) Add new source types for environmental sensors, power sensors, VM
   system events, and skew between clocks, with a sample implementation
   for each.

ok releng to go in before the branch due to the difficulty of later
pullup (widespread #ifdef removal and moved files).  Tested with release
builds on amd64 and evbarm and live testing on amd64.

Replace locking checks with uvm_page_locked_p.

extending vmem(9) to be able to allocated resources for it's own needs.
simplifying uvm_map handling (no special kernel entries anymore no relocking)
make malloc(9) a thin wrapper around kmem(9)
(with private interface for interrupt safety reasons)

releng@ acknowledged

O->A loan related statistics fixes.

make assertions simpler

remove a debug printf

- use O->A loan to serve read(2).  based on a patch from Chuck Silvers
- associated O->A loan fixes.

make lfs another pager specific flag so that it won't be affected by
an nfs hack in genfs.

Add a macro to allow a port to control from which freelists "normal" pages
can be allocated.

- fix page loaning  XXX make O->A loaning further
- add some statistics

- use mutex obj for pageable object
- add a function to wait for a mutex obj being available
- replace some "livelock" kpauses with it

cache UVM_OBJ_IS_VNODE in pqflags

redo the page clean/dirty/unknown accounting separately for file and
anonymous pages

- track the number of clean/dirty/unknown pages in the system.
- g/c PG_MARKER

remove pg->listq and uobj->memq

page cache related changes

- maintain object pages in radix tree rather than rb tree.
- reduce unnecessary page scan in putpages.  esp. when an object has a ton of
  pages cached but only a few of them are dirty.
- reduce the number of pmap operations by tracking page dirtiness more
  precisely in uvm layer.
- fix nfs commit range tracking.
- fix nfs write clustering.  XXX hack

Correct pagermap emergva allocation.  From yamt@.

Tested by building i386 kernel with DTRACE defined which died 100%.

re-arrange the end of uvm_page_recolor() to avoid the multiple exit
points.  move the call to uvm_pager_realloc_emerg() to after we
drop the uvm_fpageqlock, since it may be taken again in uvm_km_alloc().

fixes LOCKDEBUG crashes with the previous change.

Reallocate emergency pager va when ncolors is increased. (modication of
patch from mrg).

Catchup with rmind-uvmplock merge.

uvm_pagealloc_strat: fix diagnostic assert.  Reported by drochner@.

Welcome to 5.99.53!  Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
  New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
  the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
  Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
  kernel-lock on some ports).  Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

sync with head

Sync with HEAD.

Fix fencepost error.

Restore $NetBSD$

Rework page free lists to be sorted by color first rather than free_list.
Kept per color PGFL_* counter in each page free list.
Minor cleanups.

sync with head

uvm_pageidlezero: use try-lock to not occupy uvm_fpageqlock, which may
be on demand by other CPUs.  Reduces lock contention in some workloads
on many CPU (8+) systems.

Tested by tls@.

sync with head

Sync with HEAD

udpate license clauses on my code to match the new-style BSD licenses.
based on diff that rmind@ sent me.

no functional change with this commit.

udpate license clauses on my code to match the new-style BSD licenses.
based on second diff that rmind@ sent me.

no functional change with this commit.

Improve panic message for non-power of 2 page size.

Add better color matching selecting free pages.  KM pages will now allocated
so that VA and PA have the same color.  On a page fault, choose a physical
page that has the same color as the virtual address.

When allocating kernel memory pages, allow the MD to specify a preferred
VM_FREELIST from which to choose pages.  For machines with large amounts
of memory (> 4GB), all kernel memory to come from <4GB to reduce the amount
of bounce buffering needed with 32bit DMA devices.

When panicing due a non-power of 2 pagesize, include the pagesize in the
panic message.

Revert vm_physseg allocation changes.  A report says that it causes
panics when used with mplayer in heavy load.

Start merging uvm_page_physload() and uvm_page_physload_device().

Rename PGO_ZERO as PGO_HOLE, and s/uvm_page_zeropage/uvm_page_holepage/.

uvm_pglistalloc(9) returns 0 on success, not # of pages.

Don't set PG_FAKE for device pages.

XXX PG_FAKE should be renamed as PG_UNINITED!

Move zero-page into a common place, in the hope that it's shared
for other purposes.

According to Chuck Silvers, zero-page mappings don't need to be
explicitly unmapped in putpages().  Follow that advice.

... and another.

Fix build caused by a last minute change.

Be a little more friendly to dynamic physical segment registration.

Maintain an array of pointer to struct vm_physseg, instead of struct
array.  So that VM subsystem can take its pointer safely.  Pointer
to this struct will replace raw paddr_t usage in the future.

Dynamic removal is not supported yet.

Only MD data structure changes, no kernel bump needed.

Tested on i386, amd64, powerpc/ibm40x, arm11.

Oops - don't expose unnecessary data.

Fix debug code.

Abstraction fix; move physical address -> per-page metadata (struct
vm_page *) "reverse" lookup code from uvm_page.h to uvm_page.c, to
help migration to not do that.

Likewise move per-page metadata (struct vm_page *) -> physical
address "forward" conversion code into *.c too.  This is called
only low-layer VM and MD code.

Abstraction fix; move physical address -> physical segment "reverse"
lookup code from uvm_page.h to uvm_page.c.

This code is used by some pmaps to lookup per-page state (PV) from
per-segment metadata (struct vm_physseg).  This is not needed if
UVM looks up physical segment once in fault handler, then directly
passes it to pmap.  This change helps transition to that model.

The only users of vm_physseg_find() are pmap_motorola.c and
powerpc/ibm4xx/pmap.c.

Tested By:	Compiling and running powerpc/ibm4xx/pmap.c
		(evbppc/conf/OPENBLOCKS266)

s/managed device page/device page/

Use vm_physseg accessors.  Remove confusing comments.

C style; make a sentinel pointer have an exclusive value; no
functional changes.

Typo in a comment.

Minor clean up.

Minor clean up.

Remove incomplete, never worked dynamic run-time memory registration
(uvm_page_physload(9)).  This functionality will be re-added later.

Style.

Split physical device segment pages from "managed" to "managed
device".  Cache that information as a flag PG_DEVICE so that callers
don't need to walk physsegs everytime.

Remove PQ_FIXED, which means that page daemon doesn't need to know
device segment pages at all.  But still fault handlers need to know
them.

I think this is what I can do best now.

Drop the 'paddr_t avail_start' and 'paddr_t avail_end' arguments
from uvm_page_physload_device(9).

Those two arguments are used by uvm_page_physload(9) to specify a
range of physical memory available for general purpose pages (pages
which are linked to freelists).  Totally irrelevant to device
segments.

Unconditionally provide device page segment data structures and
functions as suggested by Chuck Silvers.

(Memory and device segments are being merged soon.)

Sync with HEAD (-D20101022).

sync with head

Fixes/improvements to RB-tree implementation:
1. Fix inverted node order, so that negative value from comparison operator
   would represent lower (left) node, and positive - higher (right) node.
2. Add an argument (i.e. "context"), passed to comparison operators.
3. Change rb_tree_insert_node() to return a node - either inserted one or
   already existing one.
4. Amend the interface to manipulate the actual object, instead of the
   rb_node (in a similar way as Patricia-tree interface does).
5. Update all RB-tree users accordingly.

XXX: Perhaps rename rb.h to rbtree.h, since cleaning-up..

1-3 address the PR/43488 by Jeremy Huddleston.

Passes RB-tree regression tests.
Reviewed by: matt@, christos@

Make vm_physseg lookup work when VM_PHYSSEG_MAX == 1.

Fix DIAGNOSTIC build.  Sprinkle some assertions.

sync with head.

If both __HAVE_PMAP_PHYSSEG and __HAVE_PMAP_PHYSSEG_INIT is defined,
call per-vm_physseg initialization/finalization hooks.

s/vm_physseg_find_direct/vm_physseg_find_device/

Use VM_PROT_* instead of PROT_* in UVM.

After much consideration, rename bus_space_physload_direct(9) back to
bus_space_physload_device(9).

The latter registers a segment as "device pages".  "Device pages" are
managed, but not used for general purpose memory.  Most typically XIP
pages.

Give "physseg" related functions better names.

Remove a useless assertion.

Cosmetic.

s/PG_XIP/PQ_FIXED/, meaning that the fault handler sees XIP pages as
"fixed", and doesn't pass them to paging activity.

("XIP" is a vnode specific knowledge.  It was wrong that the fault
handler had to know such a special thing.)

Rename PG_DIRECT to PG_XIP.  PG_XIP is marked to XIP vnode pages.

Reduce more diff by backing out XIP page specific code.  Allow XIP pages
to be loaned.

Mark XIP pages as PG_CLEAN and/or PG_BUSY when appropriate.  Protect
vnode lock when vm_page::flags is manipulated.

Mark XIP pages as PG_RDONLY.

Clean up; merge options DIRECT_PAGE into options XIP.

To simplify things, revert global vm_page_md hash and allocate struct
vm_page [] for XIP physical segments.

Re-define the definition of "device page"; device pages are pages of
device memory.  Pages which don't have vm_page (== can't be used for
generic use), but whose PV are tracked, are called "direct pages" from
now.

Revert partial "phys_addr" removal code.  This change is independent of
XIP, and will be done later.

sync with head

Remove an old XXX comment.  Fix a conditional build.

Sync with HEAD.

"int free_list" (VM_FREELIST_*) is specific to struct vm_page (memory
page).  Handle it only in memory physseg parts.

Record device page's properties in struct vm_physseg for future uses.
For example, framebuffers that is capable of some accelarated bus access
(e.g. write-combining) should register its capability through "int
flags".

Fix a thinko in Rev. 1.153.2.30.

Revert previous; unintended changes mixed.

Fix thinko in previous.

Fold long lines.

FIx an off-by-one in my new code.

Initial support of uvm_page_physunload(9) and uvm_page_physunload_device(9).
Note that callers of these functions are responsible to ensure that the
segment is not used.

Manage struct vm_physseg as a list, which means that struct vm_physseg
objects don't move when a segment is added / removed.

Always use struct vm_physseg *vm_physmem_ptrs[] in MD code.

Use struct vm_physseg *vm_physmem_ptrs[] in lookup code paths
(vm_physseg_find()).

Use struct vm_physseg *vm_physmem_ptrs[] in initialization code paths.

Whitespace.

Maintain not only arrays of struct vm_physseg, but also arrays of pointers
to struct vm_physseg.  This is need:

- to make the array change dynamically (unload), and

- to make the struct vm_physseg * object to be passed to device drivers as
  a cookie of a managed physical segment.

Clean up comments.

Collect a garbage.

Clean up: move memory segment specific code from uvm_page_physload_common()
to uvm_page_physload().

Remove the unfinished code to add a memory segment after uvm_page_init().
It doesn't even compile.

(In the future, we should allocate struct vm_page [] on the added memory
segment for NUMA's sake.)

Reduce memory spent on bookkeeping for large values of MAXCPUS.

Refactor uvm_page_physload_common().  Memory allocation failure here is
critical; panic if it happens.

Make uvm_page_physload() return the registered struct vm_physseg *.

Reorganise UVM locking to protect P->V state and serialise pmap(9)
operations on the same page(s) by always locking their owner.  Hence
lock order: "vmpage"-lock -> pmap-lock.

Patch, proposed on tech-kern@, from Andrew Doran.

Change struct uvm_object::vmobjlock to be dynamically allocated with
mutex_obj_alloc().  It allows us to share the locks among UVM objects.

sync with head

Don't always enable XIP on this branch to prepare the merge.  Fix build
without XIP in places.

- Use ctob() instead of ptoa() to obtain physical addresses from frame
numbers. Using ptoa() will cast to vaddr_t, which might no be adequate
for architectures where sizeof(paddr_t) > sizeof(vaddr_t) (like i386 PAE).

- small fix inside AGP heuristics to avoid masking high order bits for
systems with more than 4GB.

Reviewed by bouyer@.

See also http://mail-index.netbsd.org/tech-kern/2010/02/22/msg007373.html

Put back vm_page::phys_addr for now, because removing it involves some random
parts in the tree.  I'll revisit this after merging the branch.

Make struct vm_page_md * -> struct vm_page_md * lookup a real function and
hide its internal.  Won't cause much performance loss because results are
usually cached by callers.

Introduce uvm_page_physload_device().  This registers a physical address
range of a device, similar to uvm_page_physload() for memories.  For now,
this is supposed to be called by MD code.  We have to consider the design
when we'll manage mmap'able character devices.

Expose paddr_t -> struct vm_page * conversion function for device pages,
uvm_phys_to_vm_page_device().  This will be called by XIP vnode pager.
Because it knows if a given vnode is a device page (and its physical
address base) or not.  Don't look up device segments, but directly make a
cookie.

Enable the newly added VM_PAGE_TO_MD() only #ifdef __HAVE_VM_PAGE_MD.
Pointed out by mrg@.

Fix previous again & use VM_PAGE_TO_MD() where appropriate.

Initial MD per-page data (struct vm_page_md) lookup code for XIP'able device
pages.  Compile tested only.

Always define uvm_pageisdevice_p().  Always false if kernel is !DEVICE_PAGE.

Give new funcs better names.

Implement device page struct vm_page * handling.

Define vm_physdev / vm_nphysdev, physical address segment data for managed
device pages.

vm_nphysseg -> vm_nphysmem

Merge vm_physseg lookup routines.

Kill vm_page::phys_addr.

Abstract vm_physseg_find() to handle struct vm_page *.

Make vm_physseg lookup into a real function.

Make vm_physseg::lastpg exclusive end.

Remove JRT two line comment about cache interference since the
change specifically addresses general VIPT cache issue.  Need more
throughout comment cleanup about uvmexp.ncolor intent and significance.

uvm_pageinsert, uvm_pageremove: Pass the uboj, to/from which a pg is
inserted/removed, as an argument, because looking up a back-reference from
pg is redundant.  No functional changes.

Pass hints to uvm_pagealloc* to get it to use the right page color rather
than guess the right page color.

Add a flags argument to pmap_kenter_pa(9).
Patch showed on tech-kern@ http://mail-index.netbsd.org/tech-kern/2009/11/04/msg006434.html
No objections.

Add KASSERT(pg) to uvm_pagefree() so that invalid physical address passed
to VM_PHYS_TO_PAGE will be easily caught.

sync with head.

Move uvm_page-related DDB hooks into uvm_page.c.

Add a real API for testing if a page is a managed page, and adjust callers
to stop relying on vm_physseg_find() for this purpose.

Fix brain fart.  physmem was int not long.

Add back declaration of physmem but use the existing type (long).

Remove physmem definition to uintptr_t from another patch.

Add uvm_reclaim_hooks support for reclaiming kernel KVA space and memory.
This is used only by zfs where uvm_reclaim hook is added from arc cache.

Oked ad@.

Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.

sync with head.

Sync with HEAD.

Clarify free_list usage in uvm_page_physload() regarding faster/slower RAM.
Slower RAM should be assigned a higher free_list id.
No functional change to code, just comments and manpage

Sync with HEAD.

Pull up following revision(s) (requested by drochner in ticket #541):
	sys/uvm/uvm_page.c: revision 1.144
oops - missed a case with PMAP_PAGEIDLEZERO if md code aborts the
zeroing process, from Nicolas Joly

Pull up following revision(s) (requested by drochner in ticket #541):
	sys/uvm/uvm_page.c: revision 1.143
-fix two conditions where PQ_FREE was still/already set while the page
 was not anymore/yet on the freelist and uvm_fpageqlock was not held
-clear PQ_FREE while the page is in the works of pageidlezero
This avoids that the DMA memory allocator (pglistalloc) grabs a page
which is not on the freelist, leading to a diagnostic panic (with DEBUG)
or freelist corruption. (mostly on X server activation after a VT
switch or suspend/resume because this can allocate megabytes of AGP
memory)
This might fix PR port-i386/38989 by Alan Barrett (in case this was
a multiprocessor).

oops - missed a case with PMAP_PAGEIDLEZERO if md code aborts the
zeroing process, from Nicolas Joly

-fix two conditions where PQ_FREE was still/already set while the page
 was not anymore/yet on the freelist and uvm_fpageqlock was not held
-clear PQ_FREE while the page is in the works of pageidlezero
This avoids that the DMA memory allocator (pglistalloc) grabs a page
which is not on the freelist, leading to a diagnostic panic (with DEBUG)
or freelist corruption. (mostly on X server activation after a VT
switch or suspend/resume because this can allocate megabytes of AGP
memory)
This might fix PR port-i386/38989 by Alan Barrett (in case this was
a multiprocessor).

Sync with HEAD.

Sync with HEAD.

uvm_page_unbusy: add an assertion

Pull up following revision(s) (requested by bouyer in ticket #211):
	sys/uvm/uvm_km.c: revision 1.103
	sys/uvm/uvm_map.c: revision 1.265
	sys/uvm/uvm_page.c: revision 1.141
It's easier for kernel reserve pages to be consumed because the pagedaemon
serves as less of a barrier these days. Restrict provision of kernel reserve
pages to kmem and one of these cases:
- doing a NOWAIT allocation
- caller is a realtime thread
- caller is a kernel thread
- explicitly requested, for example by the pmap

It's easier for kernel reserve pages to be consumed because the pagedaemon
serves as less of a barrier these days. Restrict provision of kernel reserve
pages to kmem and one of these cases:

- doing a NOWAIT allocation
- caller is a realtime thread
- caller is a kernel thread
- explicitly requested, for example by the pmap

Sync with HEAD.

Sync with wrstuden-revivesa-base-2.

Sync with head.

Scale the number of kernel reserve pages by the number of CPUs.

Sync with head.

Sync with HEAD.

uvm_pageidlezero: fix a broken test which made it give up too easily.

Switch from KASSERT to CTASSERT for those asserts testing sizes of types.

Update to HEAD.

Change tree op members/typedefs to rbto_compare_* from rb_compare_*

Sync with HEAD.

Sync w/ -current. 34 merge conflicts to follow.

Sync with head.

sync with head.

- uvm_pagereplace: don't try to insert multiple pages with the same offset
  into uvm_object rbtree.
- inline static -> static inline

Sync with HEAD.

Also fix build.

Delete what appears to be a spurious assignment to an undeclared
'cpu' variable added in revision 1.133.  Restores buildability for this file.

Replace the global vm_page hash with a per vm_object rbtree.
Proposed on tech-kern@.

- vm_page: put listq, pageq into a union alongside a LIST_ENTRY, so we can
  use both types of list.

- Make page coloring and idle zero state per-CPU.

- Maintain per-CPU page freelists. When freeing, put pages onto the local
  CPU's lists and the global lists. When allocating, prefer to take pages
  from the local CPU. If none are available take from the global list as
  done now. Proposed on tech-kern@.

sync with head

Sync with HEAD.

uvm_pageidlezero:

- Use high and low water marks to try and reduce power consumption.
- Do trylock on uvm_fpageqlock, and bail if we can't get it.
- Only run on one CPU at a time.

Sync with HEAD.

sync with head.

remove a redundant pmap_update and add a comment instead.

sync with head.

sync with HEAD

sync with head.

Assert uvm_fpageqlock is held in a few more places.

sync with head.

Add some more missing pmap_update()s following pmap_kremove()s.

Sync with HEAD.

sync with head

Sync with HEAD

unwrap short lines.

sync with HEAD

Sync with HEAD

Merge vmlocking2 to head.

Sync with HEAD.

sync with head

Pull the vmlocking changes into a new branch.

Sync with HEAD.

Use atomics to maintain uvmexp.{anon,exec,file}pages.

sync with HEAD

sync with head.

Sync with HEAD.

Follow the merge of pmap.c on i386 and amd64 and move
pmap_init_tmp_pgtbl into arch/x86/x86/pmap.c. Modify the ACPI wakeup
code to restore CR4 before jumping back into kernel space as the large
page option might cover that.

sync with head.

Fix merge error.

Pad the hashlocks to 32-byte boundaries.

sync with head.

fix some races around pagedaemon and uvm_wait.  ok'ed by Andrew Doran.

Sync with HEAD.

Sync with HEAD.

Pad out the hashlocks to reduce cache traffic.

Merge unobtrusive locking changes from the vmlocking branch.

file uvm_page.c was added on branch matt-mips64 on 2007-07-21 19:21:56 +0000

Merge unobtrusive locking changes from the vmlocking branch.

Sync with head.

Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

Sync with head.

merge yamt-idlelwp branch.  asked by core@.  some ports still needs work.

from doc/BRANCHES:

	idle lwp, and some changes depending on it.

	1. separate context switching and thread scheduling.
	   (cf. gmcgarry_ctxsw)
	2. implement idle lwp.
	3. clean up related MD/MI interfaces.
	4. make scheduler(s) modular.

Note that the hashlocks must be spinlocks.

Split uvm_hashlock into an array of 32 locks.

sync with head.

Track lwp as well as proc owner with UVM_PAGE_TRKOWN

- Add two new arguments to kthread_create1: pri_t pri, bool mpsafe.
- Fork kthreads off proc0 as new LWPs, not new processes.

- Put a per-LWP lock around swapin / swapout.
- Replace use of lockmgr().
- Minor locking fixes and assertions.
- uvm_map.h no longer pulls in proc.h, etc.
- Use kpause where appropriate.

uvm_pagefree: check for already freed pages before checking the locks.

Add some lock assertions.

idlezero: don't grab the kernel lock.

Pull in the initial set of changes for the vmlocking branch.

- sync with head.
- move sched_changepri back to kern_synch.c as it doesn't know PPQ anymore.

sync with head.

TRUE -> true, FALSE -> false

Replace the Mach-derived boolean_t type with the C99 bool type.  A
future commit will replace use of TRUE and FALSE with true and false.

- separate context switching and thread scheduling.
- introduce idle lwp.
- change some related MD/MI interfaces and implement i386 version.

Merge newlock2 to head.

Sync with head.

Checkpoint work in progress.

sync with head.

merge yamt-splraiseipl branch.

	- finish implementing splraiseipl (and makeiplcookie).
	  http://mail-index.NetBSD.org/tech-kern/2006/07/01/0000.html
	- complete workqueue(9) and fix its ipl problem, which is reported
	  to cause audio skipping.
	- fix netbt (at least compilation problems) for some ports.
	- fix PR/33218.

sync with head.

put ->K loaned pages on the page queue, so that page loaning doesn't
disturb pagedaemon/pdpolicy.

Sync with head.

Checkpoint work in progress.

use workqueue for aiodoned.

sync with head

Don't inline uvm_pagealloc_pgfl().