CVS log for src/sys/kern/subr_vmem.c

Up to [cvs.NetBSD.org] / src / sys / kern

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

vmem(9): Sprinkle SET_ERROR dtrace probes.

PR kern/58378: Kernel error code origination lacks dtrace probes

vmem(9): Sort includes, remove sharp edges from macros.

No functional change intended.  (And if changing /* nothing */ to
__nothing changed anything, it was already broken!)

vmem_init(): Ensure that the quantum is a power of 2, and that if private
tags are being used, they are added to the arena before the first span is
added.

Add the notion of "private boundary tags" to vmem.  This allows vmem to
be used VERY early in boot; such consumers statically allocate the vmem
arena and boundary tags, and then explicitly add those static, private
boundary tags to the arena tag free list using the new function vmem_add_bts().

Vmem arenas that use private boundary tags will NOT consume the statically
allocated bootstrap tags used by the vmem system itself; the assumption is
that the consumer of such an arena knows what they're doing, and is responsible
for all necessary resource management.  A macro, VMEM_EST_BTCOUNT(), is
provided to help such consumers size the static boundary tag store based
on the expected number of spans and early allocations.  Once the private
tags are exhausted, the arena will dynamically allocate tags as usual.

Split the boundary tag "type" field into "type" and "flags" fields.
Initialize the flags field to 0 before inserting into an arena's free
tag list.

NFC, but makes diff for a future enhancement smaller.

bt_freetrim(): Restructure the loop as a LIST_FOREACH_SAFE() rather
than a while().  No real change in behavior now, but makes upcoming
enhancements easier.

Assert that the vmem_btag_pool has been initialized before we attempt
to allocate from it.

Add a vmem_xalloc_addr() function, which allocates a specific address
from an arena.  This is just a convenience wrapper around vmem_xalloc(),
that's just a bit more obvious how to use and performs some additional
sanity checks.

Minor changes to let this build as the "subr_vmem" test program again.

kern: KASSERT(A && B) -> KASSERT(A); KASSERT(B)

fix various typos in comments, documentation and messages.

vmem(9): Assert addresses are quantum-aligned.

fix multiplei repetitive typos in comments, messages and documentation. mainly because copy paste code big amount of files are affected.

Nix trailing whitespace.

Add vmem_xfreeall(), which frees all allocated regions in the specified arena.
All outstanding allocations MUST have been performed with vmem_xalloc() or
else the behavior is undefined.  (This also implies that the arena must also
not have a quantum cache; note this in the documentation.)

Sync with bouyer-xenpvh-base2 (HEAD)

Sync with HEAD

vmem_list_lock needs to be an adaptive lock.  Fixes LOCKDEBUG panic.

Sync with HEAD

comment

- Fix uneven performance with "bursty" vmem arenas.  Adjust locking so that
  the mutex is acquired and released only once in the happy path.  Align
  tags to cachelines.  Size the hash table according to the maximum count of
  boundary tags over the interval just gone, not the instantaneous count,
  and decay that maximum value by 50%+1 after each rehash.  Round up to the
  next power of two to eliminate divisions.  Do the rehash check unlocked.

- Hash bucket size is sizeof(vmem_hashlist), not size of a pointer to same.

Mostly merge changes from HEAD upto 20200411

Merge changes from current as of 20200406

Detangle the pagedaemon from uvm_fpageqlock:

- Have a single lock (uvmpd_lock) to protect pagedaemon state that was
  previously covered by uvmpd_pool_drain_lock plus uvm_fpageqlock.
- Don't require any locks be held when calling uvm_kick_pdaemon().
- Use uvm_free().

Add and use __FPTRCAST, requested by uwe@

Add void * function pointer casts. There are different ways to "fix" those
warnings:
    1. this one: add a void * cast (which I think is the least intrusive)
    2. add pragmas to elide the warning
    3. add intermediate inline conversion functions
    4. change the called function prototypes, adding unused arguments and
       converting some of the pointer arguments to void *.
    5. make the functions varyadic (which defeats the purpose of checking)
    6. pass command line flags to elide the warning
I did try 3 and 4 and I was not pleased with the result (sys_ptrace_common.c)
(3) added too much code and defines, and (4) made the regular use clumsy.

Typos.

update from HEAD

Pull up following revision(s) (requested by mlelstv in ticket #1521):
	share/man/man9/kmem.9: revision 1.20 via patch
	share/man/man9/vmem.9: revision 1.16
	sys/kern/subr_kmem.c: revision 1.62
	sys/kern/subr_vmem.c: revision 1.94
fix vmem_alloc() to never return an error for VM_SLEEP requests,
thus fixing kmem_alloc() to never return NULL for KM_SLEEP requests.
instead these operations will retry forever, which was the intent.

Sync with HEAD

assert that vmem_alloc() with VM_SLEEP does not fail.

Sync with HEAD

KNF. Remove extra spaces. No functional change.

Sync with HEAD

fix vmem_alloc() to never return an error for VM_SLEEP requests,
thus fixing kmem_alloc() to never return NULL for KM_SLEEP requests.
instead these operations will retry forever, which was the intent.

Sync with HEAD

to garnish, dust with _KERNEL_OPT

Rebase to HEAD as of a few days ago.

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs.  ("Protocol error: too many arguments")

sync with head

make condition for ENOMEM consistent with allocation requirement

bt_refill is and must be called with VM_NOSLEEP set, assert this
fix error path if pool_get returns NULL

Incorrect use of pointer arithmetic.

CID 1193195:  Extra sizeof expression

kill undesirable #ifndef _RUMPKERNEL

replace vmem(9) custom boundary tag allocation with a pool(9)

convert vmem, signals, powerhooks from CIRCLEQ -> TAILQ.

Turn a few __unused into __diagused

sync with HEAD

Revert the previous commit.

Always terminate qc_name with NUL because pool_init(9) uses
strcmp(3) to compare wchans.

resync from head

update comments

resync with head

printflike maintenance.

Pull up following revision(s) (requested by para in ticket #789):
	sys/kern/subr_vmem.c: revision 1.81
	sys/kern/subr_vmem.c: revision 1.77
fix a lock order reversal during global boundary tag refill.
thanks to chuq@
xxx: request pullup
Fix release of vmem_btag_lock (don't release twice in error path)

Pull up following revision(s) (requested by para in ticket #789):
	sys/kern/subr_vmem.c: revision 1.81
	sys/kern/subr_vmem.c: revision 1.77
fix a lock order reversal during global boundary tag refill.
thanks to chuq@
xxx: request pullup
Fix release of vmem_btag_lock (don't release twice in error path)

Fix release of vmem_btag_lock (don't release twice in error path)

make vmem(9) ready to be used early during bootstrap to replace extent(9)
pass memory for vmem structs into the initialization function and
do away with the static pool of vmem structs.
remove special bootstrapping of the quantum cache pools of the kmem_va_arena
as memory for pool_caches is allocated via pool_allocator_meta which is
fully operational at this point.

revert previous commit not yet fully functional, sorry

make vmem(9) ready to be used early during bootstrap to replace extent(9).
pass memory for vmem structs into the initialization functions and
do away with the static pools for this.
factor out the vmem internal structures into a private header.
remove special bootstrapping of the kmem_va_arena as all necessary memory
comes from pool_allocator_meta wich is fully operational at this point.

sync with head

fix a lock order reversal during global boundary tag refill.
thanks to chuq@
xxx: request pullup

Resync to 2012-11-19 00:00:00 UTC

sync with head

Don't use const foo const as type, one const is enough.

rework boundary-tag reserve calculation, make it more precise.
add comment about the rational behind the sizing of certain vars
used by allocation and bootstrap.
as requested by yamt@

Remove final ';' from CONDVAR_DECL macro. The caller already adds its
own.

sync with head

Pull up following revision(s) (requested by para in ticket #155):
	sys/kern/subr_vmem.c: revision 1.73
	sys/kern/subr_kmem.c: revision 1.43
	sys/rump/librump/rumpkern/vm.c: revision 1.124
make accounting for vm_inuse sane
while here don't statically allocated for more caches then required
adjust rump for static pool_cache count
should have went in with subr_vmem 1.73
don't overallocated once we leave the caches

sync to -current

make accounting for vm_inuse sane
while here don't statically allocated for more caches then required

merge to -current.

proper sizing of kmem_arena on different ports

PR port-i386/45946: Kernel locks up in VMEM system

allocate uareas and buffers from kernel_map again
add code to drain pools if kmem_arena runs out of space

do the bookkeeping right

- vmem_xalloc: remove call to uvm_kick_pdaemon() as it is not right.
- Reduce some #ifdefs.  Misc.

move condvar calls under interlock protection
call uvm_kick_pdaemon in case we can sleep and no space in arena

- Make subr_vmem.c compile as standalone again.
- Reduce some #ifdefs.

extending vmem(9) to be able to allocated resources for it's own needs.
simplifying uvm_map handling (no special kernel entries anymore no relocking)
make malloc(9) a thin wrapper around kmem(9)
(with private interface for interrupt safety reasons)

releng@ acknowledged

revert a debug printf slipped in with subr_vmem.c rev.1.64.
pointed out by Nicolas Joly.

vmem_create: copy the 'name' string rather than just keeping a reference to it.
i think it's what solaris does as at least dtrace relies on this behaviour.

Avoid return with expression in void functions.

- Replace calc_order() with ilog2(), wrap into SIZE2ORDER().
- Inline some functions, reduce some #ifdefs.

Report vmem(9) errors out-of-band so that we can use vmem(9) to manage
ranges that include the least and the greatest vmem_addr_t.  Update
vmem(9) uses throughout the kernel.  Slightly expand on the tests in
subr_vmem.c, which still pass.  I've been running a kernel with this
patch without any trouble.

Introduce a couple of new constants, VMEM_ADDR_MIN (the least possible
address in a vmem(9) arena, 0) and VMEM_ADDR_MAX (the maximum possible
address, currently 0xFFFFFFFF).  Modify several boundary conditions so
that a vmem(9) arena can allocate ranges including VMEM_ADDR_MAX.
Update documentation and tests.

These changes pass the tests in sys/kern/subr_vmem.c.  To compile the
and run the test program, run "cd sys/kern/ && gcc -DVMEM_SANITY -o
subr_vmem ./subr_vmem.c && ./subr_vmem".

comments.  related to PR/44969

sync with head

wrap a long line.

Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.

sync with head.

Sync with HEAD.

Ansify function definitions w/o arguments. Generated with sed.

Sync with HEAD.

vmem_rehash_all: remove a debug printf slipped in with the previous changes.

- fix vmem unittest.  rename VMEM_DEBUG so that it won't be abused again.
- reimplement vmem sanity checks with less code duplication.
- reimplement ddb vmem-related commands in a more consistent ways.
  remove automatic whatis.

Pull up following revision(s) (requested by ad in ticket #340):
	sys/kern/subr_vmem.c: revision 1.52
Put vm_lock into its own cache line.

bump VMEM_HASHSIZE_MAX from 8192 to 65536.

Change VMEM_HASHSIZE_INIT from 1 to 128.  This mainly benefits
quick-running or non-threaded rump jobs, where the rehash algorithm
does not have a chance to run.  For other cases it doesn't make
much difference, since the size will grow or decrease when the
rehash algorithm runs for the first time (t=10*hz currently).

Sync with HEAD.

Sync with HEAD.

Put vm_lock into its own cache line.

Update haad-dm branch to haad-dm-base2.

fix endif comment

disable VMEM_DEBUG by default.

vmem_check_sanity is just too expensive for DEBUG. Enable it only for
VMEM_DEBUG.

vmem_check_sanity: refactor overlapping check for better readability.
Move use of vmem_check_sanity into locked sections. Pointed out by ad@

ddb: make show vmem print exact span type
vmem_check_sanity: print exact span type
vmem_add1: KASSERT span types

vmem_check_sanity: fix check to correctly detect even exactly duplicate spans.

build fix: make i386 build again

Spans may never overlap. Overlapping spans mean memory corruption
when used by kmem(9), for example.
Do sanity checks to detect such spans on DEBUG kernels.

ddb: new 'show vmem' and 'show all vmems' commands.
Useful to inspect vmem(9) structures.

Sync with HEAD.

sync with head.

sync with HEAD

sync with head.

- simplify ASSERT_SLEEPABLE.
- move it from proc.h to systm.h.
- add some more checks.
- make it a little more lkm friendly.

Sync with HEAD.

sync with head.

Mark some callouts/workqueues/kthreads MPSAFE.

sync with head

sync with HEAD

Sync with HEAD

Start detangling lock.h from intr.h. This is likely to cause short term
breakage, but the mess of dependencies has been regularly breaking the
build recently anyhow.

Sync with HEAD

Sync with HEAD.

Sync with head.

vmem_whatis: print "free" btags as well.

vmem_alloc: round-up size only when necessary.  suggested by Andrew Doran.

Sync with HEAD

sync with head.

add ddb "whatis" command.  inspired from solaris ::whatis dcmd.

- separate kernel va allocation (kernel_va_arena) from
  in-kernel fault handling (kernel_map).
- add vmem bootstrap code.  vmem doesn't rely on malloc anymore.
- make kmem_alloc interrupt-safe.
- kill kmem_map.  make malloc a wrapper of kmem_alloc.

Sync with HEAD.

Sync with HEAD.

Sync with head.

sync with head

Match the docs: MUTEX_DRIVER/SPIN are now only for porting code written
for Solaris.

Sync with HEAD.

sync with head.

Sync with HEAD

Sync with HEAD.

pull subr_vmem.c rev.1.33.

sync with -HEAD

Merge from vmlocking:

- pool_cache changes.
- Debugger/procfs locking fixes.
- Other minor changes.

sync with HEAD

Sync with HEAD.

Merge scheduler changes from the vmlocking branch. All discussed on
tech-kern:

- Invert priority space so that zero is the lowest priority. Rearrange
  number and type of priority levels into bands. Add new bands like
  'kernel real time'.
- Ignore the priority level passed to tsleep. Compute priority for
  sleep dynamically.
- For SCHED_4BSD, make priority adjustment per-LWP, not per-process.

sync with head.

remove #ifdef notyet which is not necessary.

fix priorities for some kernel threads.  advised and ok'ed by Andrew Doran.

Sync with HEAD.

Follow the merge of pmap.c on i386 and amd64 and move
pmap_init_tmp_pgtbl into arch/x86/x86/pmap.c. Modify the ACPI wakeup
code to restore CR4 before jumping back into kernel space as the large
page option might cover that.

vmem_rehash_all: remove no longer unnecessary splvm/splx pairs.

sync with head.

Update for pool_cache API changes.

Sync with head.

Implementation of per-CPU work-queues support for workqueue(9) interface.
WQ_PERCPU flag for workqueue and additional argument for workqueue_enqueue()
to assign a CPU might be used. Notes:
 - For now, the list is used for workqueue_queue, which is non-optimal,
   and will be changed with array, where index would be CPU ID.
 - The data structures should be changed to be cache-friendly.

Reviewed by: <yamt>, <tech-kern>

Sync with head.

Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

periodically resize vmem hash table.

sync with head.

Sync with head.

Pullup to -current

Remove duplicate #include's
From: Slava Semushin <php-coder@altlinux.ru>

sync with head.

- Replace more simple_locks, and fix up in a few places.
- Use condition variables.
- LOCK_ASSERT -> KASSERT.

Pull in the initial set of changes for the vmlocking branch.

Sync with head.

Pass an ipl argument to pool_init/POOL_INIT to be used when initializing
the pool's lock.

Sync with HEAD.

qc_init: don't leave pointers to destroyed pools hanging or we will
allocate from them later.

- sync with head.
- move sched_changepri back to kern_synch.c as it doesn't know PPQ anymore.

sync with head.

TRUE -> true, FALSE -> false

Replace the Mach-derived boolean_t type with the C99 bool type.  A
future commit will replace use of TRUE and FALSE with true and false.

Sync with head.

sync with head.

sync with head.

Sync with head.

file subr_vmem.c was added on branch yamt-lazymbuf on 2006-12-30 20:50:06 +0000

qc_reap, qc_destroy: simplify.

vmem_destroy: don't forget to clean up qcache_t.

vmem: share qcache_t if itemsperpage is same.

protect bt_poolcache by splvm because it's shared among all arenas
including ones which can be used in interrupt context.
PR/35042 from Manuel Bouyer.

vmem_xalloc: plug memory leak on error.

- define some macros and use them.
- fix an off-by-one in testcode.

todo.

remove some __unused from function parameters.

vmem: use correct function names for ASSERT_SLEEPABLE.

qc_init: fix a problem introduced by rev.1.13.
namely, use quantum-size alignment rather than no alignment.
PR/34879 from Peter Postma.

remove a todo which has been done.

qc_init: when initializing pools, use align==1 (ie. no alignment)
rather than align==0, which is converted to ALIGN(1) by pool_init.

sync with head

vmem_alloc: use __unused rather than a cryptic
"do { if (&strat) {} } while (/* CONSTCOND */ 0);"

another day, another __unused sprinkle.

implement vmem_xalloc/xfree.
XXX importing needs some more thoughts.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

sync with head

sync with head.

file subr_vmem.c was added on branch rpaulo-netinet-merge-pcb on 2006-09-09 02:57:16 +0000

Use %zu for size_t

vmf_to_prf: tweak code to avoid a "may be used uninitialized" warning.
pointed by Kurt Schreiner.

implement kva reclamation for kmem_alloc quantum cache.

implement vmem quantum cache.

- fix integer overflows.
- bump VMEM_MAXORDER.

should fix "idx < VMEM_MAXORDER" assertion failure reported by
Martijn van Buul on current-users@.

sync with head

use ASSERT_SLEEPABLE where appropriate.

Merge from HEAD.

sync with head.

file subr_vmem.c was added on branch gdamore-uart on 2006-07-13 17:49:51 +0000

file subr_vmem.c was added on branch yamt-pdpolicy on 2006-06-26 12:52:57 +0000

fix VM_BESTFIT.

1. implement solaris-like vmem.  (still primitive, though)
2. implement solaris-like kmem_alloc/free api, using #1.
   (note: this implementation is backed by kernel_map, thus can't be
   used from interrupt context.)