CVS log for src/sys/sys/pool.h

Up to [cvs.NetBSD.org] / src / sys / sys

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

Do the last change differently:

Instead of having a pre-destruct hook, put knowledge of passive
serialization into the pool allocator directly, enabled by PR_PSERIALIZE
when the pool / pool_cache is initialized.  This will guarantee that
a passive serialization barrier will be performed before the object's
destructor is called, or before the page containing the object is freed
back to the system (in the case of no destructor).  Note that the internal
allocator overhead is different when PR_PSERIALIZE is used (it implies
PR_NOTOUCH, because the objects must remain in a valid state).

In the DRM Linux API shim, this allows us to remove the custom page
allocator for SLAB_TYPESAFE_BY_RCU.

Add pool_cache_setpredestruct(), which allows a pool cache to specify
a function to be called before the destructor for a batch of one or more
objects is called.  This can be used as a synchronization point by
subsystems that rely on the type-stable nature of pool cache objects or
subsystems that use other forms of passive serialization.

Sync with HEAD.

Add accessor functions to get the number of gets and puts on pools and
pool caches.

Sync with HEAD.

Add comments for the PR_WANTED and PR_PHINPAGE flags, remove unused
PR_LOGGING flag.

style

pool_cache:

- make all counters per-CPU and make cache layer do its work with atomic ops.
- conserve memory by caching empty groups globally.

Sync with HEAD

Sync with HEAD

slightly change and fix the semantics of pool_set*wat(), pool_sethardlimit()
and pool_prime() (and their pool_cache_* counterparts):

 - the pool_set*wat() APIs are supposed to specify thresholds for the count of
   free items in the pool before pool pages are automatically allocated or freed
   during pool_get() / pool_put(), whereas pool_sethardlimit() and pool_prime()
   are supposed to specify minimum and maximum numbers of total items
   in the pool (both free and allocated).  these were somewhat conflated
   in the existing code, so separate them as they were intended.

 - change pool_prime() to take an absolute number of items to preallocate
   rather than an increment over whatever was done before, and wait for
   any memory allocations to succeed.  since pool_prime() can no longer fail
   after this, change its return value to void and adjust all callers.

 - pool_setlowat() is documented as not immediately attempting to allocate
   any memory, but it was changed some time ago to immediately try to allocate
   up to the lowat level, so just fix the manpage to describe the current
   behaviour.

 - add a pool_cache_prime() to complete the API set.

Sync with HEAD

Avoid KASSERT(!cpu_intr_p()) when breaking into ddb and issuing

	show uvmexp

Introduce POOL_QUARANTINE, a feature that creates a window during which a
freed buffer cannot be reallocated. This greatly helps detecting
use-after-frees, because they are not short-lived anymore.

We maintain a per-pool fifo of 128 buffers. On each pool_put, we do a real
free of the oldest buffer, and insert the new buffer. Before insertion, we
mark the buffer as invalid with KASAN. On each pool_cache_put, we destruct
the object, so it lands in pool_put, and the quarantine is handled there.

POOL_QUARANTINE can be used in conjunction with KASAN to detect more
use-after-free bugs.

Kernel Heap Hardening: detect frees-in-wrong-pool on on-page pools. The
detection is already implicitly done for off-page pools.

We recycle pr_slack (unused) in struct pool, and make ph_node a union in
order to recycle an unsigned int in struct pool_item_header. Each time a
pool is created we atomically increase a global counter, and register the
current value in pp. We then propagate this value in each ph, and ensure
they match in pool_put.

This can catch several classes of kernel bugs and basically makes them
unexploitable. It comes with no increase in memory usage and no measurable
increase in CPU cost (inexistent cost actually, just one check predicted
false).

Remove POOL_SUBPAGE, it is unused, undocumented, and adds confusion.

Introduce a new flag, PR_USEBMAP, that indicates whether the pool uses a
bitmap to manage freed items. It dissociates PR_NOTOUCH from bitmaps, but
for now is set only when PR_NOTOUCH is set, which reproduces the current
behavior. Therefore, no functional change. Also clarify the code.

Introduce PR_ZERO to avoid open-coding memset()s everywhere. OK riastradh@.

Sync with HEAD, resolve a few conflicts

Add support for detecting use-after-frees in KASAN. We poison each freed
buffer, any subsequent read or write will be detected as illegal.

 * Add POOL_CHECK_MAGIC, which is disabled under KASAN, because the same
   detection is done in a better way.

 * Register the size+redzone in the pool structure, to reduce the overhead.

 * Fix the CTOR/DTOR check in KLEAK, the fields are never NULL.

Pull up following revision(s) (requested by mrg in ticket #593):
	sys/dev/marvell/mvxpsec.c: revision 1.2
	sys/arch/m68k/m68k/pmap_motorola.c: revision 1.70
	sys/opencrypto/crypto.c: revision 1.102
	sys/arch/sparc64/sparc64/pmap.c: revision 1.308
	sys/ufs/chfs/chfs_malloc.c: revision 1.5
	sys/arch/powerpc/oea/pmap.c: revision 1.95
	sys/sys/pool.h: revision 1.80,1.82
	sys/kern/subr_pool.c: revision 1.209-1.216,1.219-1.220
	sys/arch/alpha/alpha/pmap.c: revision 1.262
	sys/kern/uipc_mbuf.c: revision 1.173
	sys/uvm/uvm_fault.c: revision 1.202
	sys/sys/mbuf.h: revision 1.172
	sys/kern/subr_extent.c: revision 1.86
	sys/arch/x86/x86/pmap.c: revision 1.266 (via patch)
	sys/dev/dtv/dtv_scatter.c: revision 1.4

Allow only one pending call to a pool's backing allocator at a time.
Candidate fix for problems with hanging after kva fragmentation related
to PR kern/45718.

Proposed on tech-kern:
https://mail-index.NetBSD.org/tech-kern/2017/10/23/msg022472.html
Tested by bouyer@ on i386.

This makes one small change to the semantics of pool_prime and
pool_setlowat: they may fail with EWOULDBLOCK instead of ENOMEM, if
there is a pending call to the backing allocator in another thread but
we are not actually out of memory.  That is unlikely because nearly
always these are used during initialization, when the pool is not in
use.

Define the new flag too for previous commit.

pool_grow can now fail even when sleeping is ok. Catch this case in pool_get
and retry.

Assert that pool_get failure happens only with PR_NOWAIT.
This would have caught the mistake I made last week leading to null
pointer dereferences all over the place, a mistake which I evidently
poorly scheduled alongside maxv's change to the panic message on x86
for null pointer dereferences.

Since pr_lock is now used to wait for two things now (PR_GROWING and
PR_WANTED) we need to loop for the condition we wanted.
make the KASSERTMSG/panic strings consistent as '%s: [%s], __func__, wchan'
Handle the ERESTART case from pool_grow()

don't pass 0 to the pool flags
Guess pool_cache_get(pc, 0) means PR_WAITOK here.
Earlier on in the same context we use kmem_alloc(sz, KM_SLEEP).

use PR_WAITOK everywhere.
use PR_NOWAIT.

Don't use 0 for PR_NOWAIT

use PR_NOWAIT instead of 0

panic ex nihilo -- PR_NOWAITing for zerot

Add assertions that either PR_WAITOK or PR_NOWAIT are set.
- fix an assert; we can reach there if we are nowait or limitfail.
- when priming the pool and failing with ERESTART, don't decrement the number
  of pages; this avoids the issue of returning an ERESTART when we get to 0,
  and is more correct.
- simplify the pool_grow code, and don't wakeup things if we ENOMEM.

In pmap_enter_ma(), only try to allocate pves if we might need them,
and even if that fails, only fail the operation if we later discover
that we really do need them.  This implements the requirement that
pmap_enter(PMAP_CANFAIL) must not fail when replacing an existing
mapping with the first mapping of a new page, which is an unintended
consequence of the changes from the rmind-uvmplock branch in 2011.

The problem arises when pmap_enter(PMAP_CANFAIL) is used to replace an existing
pmap mapping with a mapping of a different page (eg. to resolve a copy-on-write).
If that fails and leaves the old pmap entry in place, then UVM won't hold
the right locks when it eventually retries.  This entanglement of the UVM and
pmap locking was done in rmind-uvmplock in order to improve performance,
but it also means that the UVM state and pmap state need to be kept in sync
more than they did before.  It would be possible to handle this in the UVM code
instead of in the pmap code, but these pmap changes improve the handling of
low memory situations in general, and handling this in UVM would be clunky,
so this seemed like the better way to go.

This somewhat indirectly fixes PR 52706, as well as the failing assertion
about "uvm_page_locked_p(old_pg)".  (but only on x86, various other platforms
will need their own changes to handle this issue.)
In uvm_fault_upper_enter(), if pmap_enter(PMAP_CANFAIL) fails, assert that
the pmap did not leave around a now-stale pmap mapping for an old page.
If such a pmap mapping still existed after we unlocked the vm_map,
the UVM code would not know later that it would need to lock the
lower layer object while calling the pmap to remove or replace that
stale pmap mapping.  See PR 52706 for further details.
hopefully workaround the irregularly "fork fails in init" problem.
if a pool is growing, and the grower is PR_NOWAIT, mark this.
if another caller wants to grow the pool and is also PR_NOWAIT,
busy-wait for the original caller, which should either succeed
or hard-fail fairly quickly.

implement the busy-wait by unlocking and relocking this pools
mutex and returning ERESTART.  other methods (such as having
the caller do this) were significantly more code and this hack
is fairly localised.
ok chs@ riastradh@

Don't release the lock in the PR_NOWAIT allocation. Move flags setting
after the acquiring the mutex. (from Tobias Nygren)
apply the change from arch/x86/x86/pmap.c rev. 1.266 commitid vZRjvmxG7YTHLOfA:

In pmap_enter_ma(), only try to allocate pves if we might need them,
and even if that fails, only fail the operation if we later discover
that we really do need them.  If we are replacing an existing mapping,
reuse the pv structure where possible.

This implements the requirement that pmap_enter(PMAP_CANFAIL) must not fail
when replacing an existing mapping with the first mapping of a new page,
which is an unintended consequence of the changes from the rmind-uvmplock
branch in 2011.

The problem arises when pmap_enter(PMAP_CANFAIL) is used to replace an existing
pmap mapping with a mapping of a different page (eg. to resolve a copy-on-write).
If that fails and leaves the old pmap entry in place, then UVM won't hold
the right locks when it eventually retries.  This entanglement of the UVM and
pmap locking was done in rmind-uvmplock in order to improve performance,
but it also means that the UVM state and pmap state need to be kept in sync
more than they did before.  It would be possible to handle this in the UVM code
instead of in the pmap code, but these pmap changes improve the handling of
low memory situations in general, and handling this in UVM would be clunky,
so this seemed like the better way to go.

This somewhat indirectly fixes PR 52706 on the remaining platforms where
this problem existed.

hopefully workaround the irregularly "fork fails in init" problem.

if a pool is growing, and the grower is PR_NOWAIT, mark this.
if another caller wants to grow the pool and is also PR_NOWAIT,
busy-wait for the original caller, which should either succeed
or hard-fail fairly quickly.

implement the busy-wait by unlocking and relocking this pools
mutex and returning ERESTART.  other methods (such as having
the caller do this) were significantly more code and this hack
is fairly localised.

ok chs@ riastradh@

update from HEAD

add two new members to uvmexp_sysctl{}:  bootpages and poolpages.
bootpages is set to the pages allocated via uvm_pageboot_alloc().
poolpages is calculated from the list of pools nr_pages members.

this brings us closer to having a valid total of pages known by
the system, vs actual pages originally managed.

XXX: poolpages needs some handling for PR_RECURSIVE pools still.

Define the new flag too for previous commit.

XXX pullup-8
XXX pullup-7
XXX pullup-6
XXX pullup-5...

Sync with HEAD

move stdbool.h to sys, reuire pool.h to include <sys/stdbool.h> for bool,
centralize definitions of bool, true, false, to <sys/stdbool.h>

Introduce POOL_REDZONE.

Don't nest structure definitions.

Rebase to HEAD as of a few days ago.

Rebase.

Add kern.pool for memory pool stats.

sync with head

Now that pool_cache_invalidate() is synchronous and can handle per-CPU
caches, merge together pool_drain_start() and pool_drain_end() into

bool pool_drain(struct pool **ppp);

"bool" value indicates whether reclaiming was fully done (true) or not (false)
"ppp" will contain a pointer to the pool that was drained (optional).

See http://mail-index.netbsd.org/tech-kern/2012/06/04/msg013287.html

sync to latest -current.

sync with head.

G/C POOL_DIAGNOSTIC option.  No objection on tech-kern@.

sync with head

merge to -current.

extending vmem(9) to be able to allocated resources for it's own needs.
simplifying uvm_map handling (no special kernel entries anymore no relocking)
make malloc(9) a thin wrapper around kmem(9)
(with private interface for interrupt safety reasons)

releng@ acknowledged

change printf gcc attribute to __printflike(), requested by joerg.

add more missing printf attributes.

Sync with HEAD.

sync with head.

sync with head

Report result of pool_reclaim() from pool_drain_end().

Uses MAXCPUS from sys/param.h; include that explicitly instead of by
accident.

sync with head

Turn PA_INITIALIZED to a reference count for the pool allocator, and once
it drops to zero destroy the mutex we initialize. This fixes the problem
mentioned in

	http://mail-index.netbsd.org/tech-kern/2009/12/28/msg006727.html

Also remove pa_flags now that it's no longer needed.

Idea from matt@, okay matt@.

- pool_cache_invalidate(): broadcast a cross-call to drain the per-CPU
  caches before draining the global cache.
- pool_cache_invalidate_local(): remove.

Add pool_cache_invalidate_local() to the pool_cache(9) API, to permit
per-CPU objects invalidation when cached in the pool cache.

See http://mail-index.netbsd.org/tech-kern/2009/10/05/msg006206.html .

Reviewed by bouyer@. Thanks!

sync with head

Wipe out the last vestiges of POOL_INIT with one swift stroke.  In
most cases, use a proper constructor.  For proplib, give a local
equivalent of POOL_INIT for the kernel object implementation.  This
way the code structure can be preserved, and a local link set is
not hazardous anyway (unless proplib is split to several modules,
but that'll be the day).

tested by booting a kernel in qemu and compile-testing i386/ALL

sync with head.

Sync with HEAD.

Sync with wrstuden-revivesa-base-2.

Sync with head.

- Keep cache locked while allocating a cache group - later we might want
  to automatically tune the group sizes at run time.
- Fix broken assertion.
- Avoid another test+branch.

Sync with HEAD.

sync with head.

sync with head.

Remove clause 3 and 4 from TNF licenses

Sync with HEAD.

sync with head

sync with HEAD

Sync with HEAD

Sync with HEAD.

Sync with head.

Merge more changes from vmlocking2, mainly:

- Locking improvements.
- Use pool_cache for more items.

Make values of PR_WAITOK/PR_NOWAIT match KM_SLEEP/KM_NOSLEEP. This makes it
trivial to emulate most of Solaris' kmem_cache interface with a few macros.

- Support two different sizes of pool_cache group. The default has 14 or 15
  items, and the new large groups (for busy caches) have 62 or 63 items.
- Add PR_LARGECACHE flag as a hint that a pool_cache should use large groups.
  This should be eventually be tuned at runtime.
- Report group size for vmstat -C.

Sort list of pools/caches to make easier them easier to find.

Sync with HEAD

sync with head.

add ddb "whatis" command.  inspired from solaris ::whatis dcmd.

add pool_cache_bootstrap_destroy.  will be used by vmem.

sync with head

Sync with HEAD.

Sync with HEAD.

Sync with HEAD

Work around issues with pool_cache on sparc.

sync with head.

Sync with HEAD

Sync with HEAD.

sync with -HEAD

Merge from vmlocking:

- pool_cache changes.
- Debugger/procfs locking fixes.
- Other minor changes.

- Use a cross call to drain the per-CPU component of pool caches.
- When draining, skip over pools that are completly inactive.

sync with head.

- Add a CPU layer to pool caches. In combination with vmem/kmem this
  provides CPU-local slab/object and general purpose allocators. The
  strategy used is as described in Jeff Bonwick's USENIX paper, except in
  at least one place where the described allocation strategy doesn't make
  sense. For exclusive access to the CPU layer the IPL is raised or kernel
  preemption disabled. Where the interrupt priority levels are software
  emulated this is much cheaper than taking a lock, and I think that
  writing to a local %pil register is likely to have a similar penalty to
  taking a lock.

  No tuning of the group sizes is currently done - all groups have 15
  items each, but this should be fairly easy to implement. Also, the
  reclamation mechanism should probably use a cross-call to drain the
  CPU-level caches on remote CPUs.

  Currently this causes kernel memory corruption on i386, yet works without
  a problem on amd64. The cache layer is disabled for the time being until I
  can find the bug.

- Change the pool_cache API so that the caches are themselves dynamically
  allocated, and that each cache is tied to a single pool only. Add some
  stubs to change pool_cache parameters that call directly through to the
  pool layer (e.g. pool_cache_sethiwat). The idea here is that pool_cache
  should become the default object allocator (and so LKM friendly), and
  that the pool allocator should be for kernel-internal use only. This will
  be posted to tech-kern@ for review.

sync with head.

Pull in the initial set of changes for the vmlocking branch.

Sync with head.

Pass an ipl argument to pool_init/POOL_INIT to be used when initializing
the pool's lock.

sync with head.

Merge newlock2 to head.

Add some DEBUG code to check that items being freed were previously
allocated from the same source. Needs to be enabled via DDB.

sync with head.

sync with head

sync with head.

implement PR_NOALIGN.  (allow unaligned pages)
to be used by vmem quantum cache.

Merge from HEAD.

Back out previous change.  The problem was not here.  Oops.

Do not define the link_set symbol as static, or a.out linker won't find it.

sync with head.

sync with head.

Sync with head.

Sync with head.

remove a nested function so that its initializers can be compiled by gcc2.
pointed by Havard Eidnes.

Expose struct pool::struct callback_entry pr_reclaimerentry only when _KERNEL
is defined.  Fix build of src/sbin/mount/mount_tmpfs/mount_tmpfs.c.

move wait points for kva from upper layers to vm_map.  PR/33185 #1.

XXX there is a concern about interaction with kva fragmentation.
see: http://mail-index.NetBSD.org/tech-kern/2006/05/11/0000.html

Sync with head.

Pull up following revision(s) (requested by bjh21 in ticket #1192):
	sys/sys/pool.h: revision 1.48
	sys/kern/subr_pool.c: revision 1.112
Medium-sized overhaul of POOL_SUBPAGE support so that:
1: I can understand it, and
2: It works.
Notable externally-visible changes are that POOL_SUBPAGE now has to be a
compile-time constant, and that trying to initialise a pool whose objects are
larger than POOL_SUBPAGE automatically generates a pool that doesn't use
subpages.
NetBSD/acorn26 now boots multi-user again.

Pull up following revision(s) (requested by bjh21 in ticket #1192):
	sys/sys/pool.h: revision 1.48
	sys/kern/subr_pool.c: revision 1.112
Medium-sized overhaul of POOL_SUBPAGE support so that:
1: I can understand it, and
2: It works.
Notable externally-visible changes are that POOL_SUBPAGE now has to be a
compile-time constant, and that trying to initialise a pool whose objects are
larger than POOL_SUBPAGE automatically generates a pool that doesn't use
subpages.
NetBSD/acorn26 now boots multi-user again.

sync with head.

Medium-sized overhaul of POOL_SUBPAGE support so that:
1: I can understand it, and
2: It works.
Notable externally-visible changes are that POOL_SUBPAGE now has to be a
compile-time constant, and that trying to initialise a pool whose objects are
larger than POOL_SUBPAGE automatically generates a pool that doesn't use
subpages.

NetBSD/acorn26 now boots multi-user again.

Sync with head.

add "show all pools" command for ddb.

Sync with HEAD. Here we go again...

optimize pool_caches similarly to how I optimized pools before:
split the single list of pool cache groups into three lists:
completely full, partially full, and completely empty.
use LIST instead of TAILQ where appropriate.

Sync with HEAD.

introduce a new flag for pool_init, PR_NOTOUCH.
if it's specified, don't use free items as storage for internal state.
so that we can use pools for non memory backed objects.
inspired from solaris's KMC_NOTOUCH.

Fix the sync with head I botched.

Sync with HEAD.

Sync with HEAD

Pull up revision 1.44 (requested by thorpej in ticket #522):
Remove PR_IMMEDRELEASE, since setting the high water mark will achieve
the same thing.
Pointed out back in January by YAMAMOTO Takashi.

Remove PR_IMMEDRELEASE, since setting the high water mark will achieve
the same thing.

Pointed out back in January by YAMAMOTO Takashi.

Initialise (most) pools from a link set instead of explicit calls
to pool_init.  Untouched pools are ones that either in arch-specific
code, or aren't initialiased during initial system startup.

 Convert struct session, ucred and lockf to pools.

Add a new pool initialization flag, PR_IMMEDRELEASE.  This flag causes
idle pool pages to be returned to the system immediately upon becoming
de-fragmented.

Also, in pool_do_put(), don't free back an idle page unless we are over
our minimum page claim.

two changes in improve scalability:

 (1) split the single list of pages allocated to a pool into three lists:
     completely full, partially full, and completely empty.
     there is no longer any need to traverse any list looking for a
     certain type of page.

 (2) replace the 8-element hash table for out-of-page page headers
     with a splay tree.

these two changes (together with the recent enhancements to the wait code)
give us linear scaling for a fork+exit microbenchmark.

remove variable names from prototype for consistency.

Add the ability for pool caches to cache the physical address of
objects.  Clients of the pool_cache API must consistently use
the "paddr" variants or not, otherwise behavior is undefined.

Enable this on Alpha, ARM, MIPS, and x86.  Other platforms must
define POOL_VTOPHYS() in the appropriate manner in order to enable
the feature.

Part 1 of a series of simple patches contributed by Wasabi Systems
to improve network performance.

sync kqueue branch with HEAD

catch up with -current.

Catch up to -current.

Fix signed/unsigned comparison warnings from GCC 3.3.

Catch up to -current.
(CVS: It's not just a program. It's an adventure!)

Catch up with -current.

Move 'struct pool_cache_group' definition into <sys/pool.h>

Fix indentation.

Fix function decl. of _pool_reclaim() so that subr_pool.c compiles again
even if POOL_DIAGNOSTIC is defined.

Put back pool_prime(); the i386 mp pmap uses it.

Remove pool_prime().  Nothing uses it, and how it should be used it not
really well-defined in the absense of PR_STATIC.

Remove PR_MALLOCOK and PR_STATIC.  The former wasn't actually used,
and the latter, while there was some code tested the bit, was woefully
incomplete and also unused by anything.  Besides, PR_STATIC functionality
could be better handled by backend allocators anyhow.

From art@openbsd.org

Add an optional "drain" client callback, which can be set by the new
pool_set_drain_hook().  This hook is called in three cases:
* When a pool has hit the hard limit, just before either erroring
  out or sleeping.
* When a backend allocator fails to allocate memory.
* Just before trying to reclaim pages in pool_reclaim().

This hook requests the client to try and free some items back to
the pool.

From art@openbsd.org.

Remove PR_FREEHEADER; nothing uses it anymore.

From art@openbsd.org.

Pool deals fairly well with physical memory shortage, but it doesn't
deal with shortages of the VM maps where the backing pages are mapped
(usually kmem_map).  Try to deal with this:

* Group all information about the backend allocator for a pool in a
  separate structure.  The pool references this structure, rather than
  the individual fields.
* Change the pool_init() API accordingly, and adjust all callers.
* Link all pools using the same backend allocator on a list.
* The backend allocator is responsible for waiting for physical memory
  to become available, but will still fail if it cannot callocate KVA
  space for the pages.  If this happens, carefully drain all pools using
  the same backend allocator, so that some KVA space can be freed.
* Change pool_reclaim() to indicate if it actually succeeded in freeing
  some pages, and use that information to make draining easier and more
  efficient.
* Get rid of PR_URGENT.  There was only one use of it, and it could be
  dealt with by the caller.

From art@openbsd.org.

Sync kqueue branch with -current.

Catch up to -current.

Catch up with -current. Fix some bogons in the sparc64 kbd/ms
attach code. cd18xx conversion provided by mrg.

Add support for allocating pool memory in units smaller than a whole page.
This is activated by defining POOL_SUBPAGE to the size of the new allocation
unit, and makes pools much more efficient on machines with obscenely large
pages.  It might even make four-megabyte arm26 systems usable.

Catch up to -current.

Jason's last fix broke builds with POOL_DIAGNOSTIC (the kernel didn't link
as pool_get was never defined).  Fix that.

Do the reentrancy checking if POOL_DIAGNOSTIC, not DIAGNOSTIC.  Prevents
ABI change for diagnostic vs. non-diagnostic kernels.

Make this build again ifdef DIAGNOSTIC (oops)

Remove pool reentrancy testing overhead unless DIAGNOSTIC is defined.
Previously, we passed __FILE__ and __LINE__ on all pool_get/pool_set calls.

This change results in a measured 1.2% performance improvement in
ping-flood packets-per-second as reported by ping(8).

Rearrange the code that adds pages of objects to the pool; require
that the caller allocate the pool_item_header when it allocates the
pool page, so we can avoid a locking pitfall (sleeping with a simple
lock held).

Also revive pool_prime(), as there are some letigimate uses of it,
but in doing so, eliminate some of the bogosities of the old version
(i.e. don't do an implicit "setlowat", just prime the pool, and incr
the minpages for each additional page we add, and compute the number
of pages to prime in a way that callers would expect).

Remove pool_create() and pool_prime().  Nothing except pool_create()
used pool_prime(), and no one uses pool_create() anymore.

This makes it easier to fix a locking pitfall.

Add pool_cache_destruct_object(), used to force destruction of
an object and release back into the pool.

Sync with HEAD (for UBC fixes).

Add some basic statistics to pool_cache.

Sync with HEAD.

This is a first-cut implementation of support for caching of
constructed objects in the pool allocator, similar to caching
of constructed objects in the Solaris SLAB allocator.

This implementation is a separate API (pool_cache_*()) layered
on top of pools to keep the caching complexity out of the way
of pools that won't benefit from it.

While we're here, allow pool items to be as large as the pool
page size.

ANSI'ify.

Update thorpej_scsipi to -current as of a month ago

Change ratecap to its presumably intended type of struct timeval.

Create new pool flag PR_LIMITFAIL, indicating that even PR_WAIT
allocations should fail if the pool is at its hard limit.
Document flag in pool(9).
Use it in mbuf.h for the first allocate call for M_GET, M_GETHDR, and
MCLGET, so that m_reclaim gets called even for blocking allocations.

Sync w/ -current.

Improve the pool allocator's diagnostic helpers, adding the ability to
log on a per-pool basis, reentrancy checking, and dumping various pool
information from DDB.

Yet more fixes to the pool allocator:

- Protect userspace from unnecessary header inclusions (as noted on
current-users).

- Some const poisioning.

- GREATLY simplify the locking protocol, and fix potential deadlock
scenarios.  In particular, assume that the back-end page allocator
provides its own locking mechanism (this is currently true for all
such allocators in the NetBSD kernel).  Doing so allows us to simply
use one spin lock for serialized access to all r/w members of the pool
descriptor.  The spin lock is released before calling the back-end
allocator, and re-acquired upon return from it.

- Fix a problem in pr_rmpage() where a data structure was referenced
after it was freed.

- Minor tweak to page manaement.  Migrate both idle and empty pages
to the end of the page list.  As soon as a page becomes un-empty
(by a pool_put()), place it at the head of the page list, and set
curpage to point to it.  This reduces fragmentation as well as the
time required to find a non-empty page as soon as curpage becomes
empty again.

- Use mono_time throughout, and protect access to it w/ splclock().

- In pool_reclaim(), if freeing an idle page would reduce the number
of allocatable items to below the low water mark, don't.

Fix several bugs/deficiencies in the pool allocator:

- Add support for hard limits, with optional rate-limited logging of
a warning message when the pool limit is reached.  (This will be used
to fix a bug in mbuf cluster allocation on the MIPS and Alpha ports.)

- Fix some locking protocol errors.  This required splitting pr_flags
into pr_flags (which is protected by the spin lock) and pr_roflags (which
are `read only' flags, set when the pool is initialized, and never changed
again; these do not need to be protected by a mutex).

- Make the low water support actually mean something.  When a low water
mark is set, add free items to the pool until the low water mark is
reached.  When an item allocation causes the number of free items to
drop below the low water mark, make the pool catch up to it.  This can
make the pool allocator more useful for several applications (e.g.
pmap `pv entry' management) and more robust for others (for e.g. mbuf
and mbuf cluster allocation, so that the pagedaemon can use NFS to clean
pages on diskless systems without completely running dry on buffers to
receive packets in during extreme memory shoratages).

- Add a comment where we sleep waiting for more pages for the back-end
page allocator.  Specifically, instead of sleeping potentially forever,
perhaps we should just wake up once a second to try allocating a page
again.  XXX Revisit this soon.

Make this compile with POOL_DIAGNOSTIC, and add a POOL_LOGSIZE option.
Defopt these.

Prototype pool_print() and pool_chk() if DEBUG.
Initialize pool hash table with PR_HASHTABSIZE (i.e., 8) LIST_INIT()s
instead of one memset().
Only check for page != ph->ph_page if PR_PHINPAGE is set (in pool_chk()).
Print pool base pointer when reporting page inconsistency in pool_chk().

In addition to the spinlock, use the lockmgr() to serialize access to
the back-end page allocator. This allows the back-end to sleep since we
now relinquish the spin lock after acquiring the long-term lock.

Fix copyright typos...

Update copyright.

Add an alternate pool page allocator that can be used if the pool is
never accessed in interrupt context.  In the UVM case, this uses the
kernel_map, to reduce usage of the previous kmem_map resource.

Revert cdevsw mmap routines to return int.

Instrument "idle pages" (i.e. pages which have no items allocated from
them, and could thus be freed back to the system).

Add PR_NOWAIT (0) for symmetry.

Re-vamped pool manager.
	* support for customized memory supplier
	* automatic page reclaim by VM system
	* time-based hysteresis
	* cache coloring (after Bonwick's "slabs")

Add option to use "static" storage provided by the caller.
From Matthias Drochner.

add/cleanup multiple inclusion protection.

Memory pool resource utility.