CVS log for src/sys/kern/kern_mutex.c

Up to [cvs.NetBSD.org] / src / sys / kern

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

kern_mutex.c: Sort includes.  No functional change intended.

sys/lwp.h: Nix sys/syncobj.h dependency.

Remove it in ddb/db_syncobj.h too.

New sys/wchan.h defines wchan_t so that users need not pull in
sys/syncobj.h to get it.

Sprinkle #include <sys/syncobj.h> in .c files where it is now needed.

- Simplify how priority boost for blocking in kernel is handled.  Rather
  than setting it up at each site where we block, make it a property of
  syncobj_t.  Then, do not hang onto the priority boost until userret(),
  drop it as soon as the LWP is out of the run queue and onto a CPU.
  Holding onto it longer is of questionable benefit.

- This allows two members of lwp_t to be deleted, and mi_userret() to be
  simplified a lot (next step: trim it down to a single conditional).

- While here, constify syncobj_t and de-inline a bunch of small functions
  like lwp_lock() which turn out not to be small after all (I don't know
  why, but atomic_*_relaxed() seem to provoke a compiler shitfit above and
  beyond what volatile does).

Remove dodgy and unused mutex_owner_running() & rw_owner_running().

Pull up following revision(s) (requested by riastradh in ticket #1860):

	sys/kern/kern_rwlock.c: revision 1.67
	sys/kern/kern_lock.c: revision 1.182
	sys/kern/kern_mutex.c: revision 1.102
	(all via patch)

Sprinkle __predict_{true,false} for panicstr checks

Pull up following revision(s) (requested by riastradh in ticket #1677):

	sys/kern/kern_rwlock.c: revision 1.67
	sys/kern/kern_lock.c: revision 1.182
	sys/kern/kern_mutex.c: revision 1.102

Sprinkle __predict_{true,false} for panicstr checks

Pull up following revision(s) (requested by riastradh in ticket #265):

	sys/kern/kern_rwlock.c: revision 1.67
	sys/kern/kern_lock.c: revision 1.182
	sys/kern/kern_mutex.c: revision 1.102

Sprinkle __predict_{true,false} for panicstr checks

kern: New struct syncobj::sobj_name member for diagnostics.

XXX potential kernel ABI change -- not sure any modules actually use
struct syncobj but it's hard to rule that out because sys/syncobj.h
leaks into sys/lwp.h

mutex(9): Write comments in terms of ordering semantics.

Phrasing things in terms of implementation details like `acquiring
and locking cache lines' both suggests a particular cache coherency
protocol, paints an incomplete picture for more involved protocols,
and doesn't really help to prove theorems the way ordering relations
do.

No functional change intended.

mutex(9): Omit needless membar_consumer.

In practical terms, this is not necessary because MUTEX_SET_WAITERS
already issues MUTEX_MEMBAR_ENTER, which on all architectures is a
sequential consistency barrier, i.e., read/write-before-read/write,
subsuming membar_consumer.

In theoretical terms, MUTEX_MEMBAR_ENTER might imply only
write-before-read/write, so one might imagine that the
read-before-read ordering of membar_consumer _could_ be necessary.
However, the memory operations that are significant here are:

1. load owner := mtx->mtx_owner
2. store mtx->mtx_owner := owner | MUTEX_BIT_WAITERS
3. load owner->l_cpu->ci_curlwp to test if equal to owner

(1) is program-before (2) and at the same memory location,
mtx->mtx_owner, so (1) happens-before (2).

And (2) is separated in program order by MUTEX_MEMBAR_ENTER from (3),
so (2) happens-before (3).

So even if the membar_consumer were intended to guarantee that (1)
happens-before (3), it's not necessary, because we can already prove
it from MUTEX_MEMBAR_ENTER.

But actually, we don't really need (1) happens-before (3), exactly;
what we really need is (2) happens-before (3), since this is a little
manifestation of Dekker's algorithm between cpu_switchto and
mutex_exit, where each CPU sets one flag and must ensure it is
visible to the other CPUs before testing the other flag -- one flag
here is the MUTEX_BIT_WAITERS bit, and the other `flag' here is the
condition owner->l_cpu->ci_curlwp == owner; the corresponding logic,
in cpu_switchto, is:

1'. store owner->l_cpu->ci_curlwp := owner
2'. load mtx->mtx_owner to test if MUTEX_BIT_WAITERS set

kern: Nix mutex_owner.

There is no valid reason to use this except in assertions of the form

	KASSERT(mutex_owner(lock) == curlwp),

which is more obviously spelled as

	KASSERT(mutex_owned(lock)).

Exception: There's one horrible kludge in zfs that abuses this, which
should be eliminated.

XXX kernel revbump -- deleting symbol

PR kern/47114

mutex(9): Simplify membars.

- Elide macro indirection for membar_acquire.
- Use atomic_store_release instead of membar_release and store.

No functional change intended.  Possible very very very very minor
performance gain on architectures with a native store-release
instruction.

Note: It is possible there are some code paths that worked by
accident with this change which could, in theory, break now, such as
the logic I recently fixed in kern_descrip.c that assumed a
mutex_enter/exit cycle would serve as a store-before-store barrier:

	fp->f_... = ...;		// A

	/* fd_affix */
	mutex_enter(&fp->f_lock);
	fp->f_count++;
	mutex_exit(&fp->f_lock);
	...
	ff->ff_file = fp;		// B

This logic was never correct, and is likely already broken in
practice on aarch64 because the mutex_exit stub already uses STLXR
instead of DMB ISH(ST); STXR.  This change only affects the slow path
of mutex_exit, so it doesn't change even the accidental guarantees
mutex_exit makes in all paths.

KERNEL_LOCK(9): Minor tweaks to ci->ci_biglock_wanted access.

1. Use atomic_load_relaxed to read ci->ci_biglock_wanted from another
   CPU, for clarity and to avoid the appearance of data races in thread
   sanitizers.  (Reading ci->ci_biglock_wanted on the local CPU need
   not be atomic because no other CPU can be writing to it.)

2. Use atomic_store_relaxed to update ci->ci_biglock_wanted when we
   start to spin, to avoid the appearance of data races.

3. Add comments to explain what's going on and cross-reference the
   specific matching membars in mutex_vector_enter.

related to PR kern/57240

Sprinkle __predict_{true,false} for panicstr checks

Simplify. Same code before and after.

mutex(9): Properly declare _mutex_init in sys/mutex.h.

mutex(9): Convert to membar_acquire/release.

Except for setting the waiters bit -- not sure if this is actually
required to be store-before-load/store.  Seems unlikely -- surely
we'd have seen some serious bug by now if not, because membar_enter
has failed to guarantee that on x86! -- but I'm leaving it for now
until I have time to think enough about it to be sure one way or
another.

- In kern_mutex.c, if MUTEX_CAS() is not defined, define it in terms of
  atomic_cas_ulong().
- For arm, ia64, m68k, mips, or1k, riscv, vax: don't define our own
  MUTEX_CAS(), as they either use atomic_cas_ulong() or equivalent
  (atomic_cas_uint() on m68k).
- For alpha and sparc64, don't define MUTEX_CAS() in terms of their own
  _lock_cas(), which has its own memory barriers; the call sites in
  kern_mutex.c already have the appropriate memory barrier calls.  Thus,
  alpha and sparc64 can use default definition.
- For sh3, don't define MUTEX_CAS() in terms of its own _lock_cas();
  atomic_cas_ulong() is strong-aliased to _lock_cas(), therefore defining
  our own MUTEX_CAS() is redundant.

Per thread:
	https://mail-index.netbsd.org/tech-kern/2021/07/25/msg027562.html

Sync with HEAD.

Sync with HEAD.

Fix an IPI deadlock scenario that resulted in a TLB shootdown timeout
panic reported by John Klos on port-alpha:

- pmap_tlb_shootnow(): If we acquire a pmap's activation lock, we will
  have raised the IPL on the current CPU to IPL_SCHED until we drop
  the tlb_lock (due to how nested spin mutexes work).  As such, when
  we release the activation lock, forcibly lower our IPL back to IPL_VM
  so that we can receive and process IPIs while waiting for other CPUs
  to process the shootdowns.
- mutex_vector_enter(): Invoke SPINLOCK_SPIN_HOOK while spinning to acquire
  a spin mutex.  This is a nop on most platforms, but it's important on
  the Alpha.  Without this, IPIs (and thus TLB shootdowns) cannot be
  processed if trying to acquire an IPL_SCHED spin mutex such as those
  used by the scheduler.

...and while we're poking around in here:

- Rework the Alpha SPINLOCK_SPIN_HOOK to only check curcpu()->ci_ipis
  if the current CPU's IPL is >= IPL_CLOCK (thus ensuring that preemption
  is disabled and thus guaranteeing that curcpu() is stable).  (Alpha does
  not yet support kernel preemption, but this is now one less thing that
  would need to be fixed.)

Consistently right-justify backslash in macro definition.
No binary changes.

Sync w/ HEAD.

it's cpu_switchto (not cpu_switch)

Fixup the big mutex_exit comment a little. It's actually, cpu_switchto
that posts a store fence and it's AFTER setting curlwp.

More to come.

Trailing whitespace

PR kern/55251 (use of ZFS may trigger kernel memory corruption (KASAN error))

Previous wasn't quite right.  Redo it differently - disable preemption
earlier instead.

PR kern/55251: use of ZFS may trigger kernel memory corruption

mutex_vector_enter(): reload mtx_owner with preemption disabled before
calling mutex_oncpu(), otherwise lwp_dtor() can intervene.

Merge changes from current as of 20200406

Pull up following revision(s) (requested by chs in ticket #768):

	sys/kern/kern_mutex.c: revision 1.90

split an "a && b" assertion into two so it's clear in the dump which condition
was not true even if both are true by the time the dump is written.

split an "a && b" assertion into two so it's clear in the dump which condition
was not true even if both are true by the time the dump is written.

Sync with head.

Update a comment.

hppa has custom adaptive mutexes.  Allow it to build again while not
reintroducing the main read of mtx_owner that I wanted to eliminate.

mutex_vector_enter(): avoid some unneeded reads of mtx_owner.

Comment on previous explaining why it's needed.

mutex_vector_exit: if the arch doesn't have assembly stubs, we need to
unconditionally release the lock if(cold), so mutexes can be used before
interrupts are set up.  Removed with panicstr checks in 1.81.

Balance the parentheses - hopefully unbreak the build.

Inverted test.

Fix build break.

- Add a mutex_owner_running() for the benefit of the pagedaemon, which
  needs help with locking things in reverse order.
- Expunge the panicstr checks.
- Make MUTEX_NODEBUG work for adaptive mutexes too.

Get rid of MUTEX_RECEIVE/MUTEX_GIVE.

Sync with HEAD

Avoid prepending a timestamp to lock debug outputs on ddb

Lock printer functions (lockops_t#lo_dump) use printf_nolog to print, but
printf_nolog now prepends a timestamp which is unnecessary for ddb:

    db{0}> show all locks/t
    [Locks tracked through LWPs]
    Locks held by an LWP (iperf):
    Lock 0 (initialized at soinit)
    lock address : 0xffffedeb84b06080 type     :     sleep/adaptive
    initialized  : 0xffffffff806d8c3f
    shared holds :                  0 exclusive:                  1
    shares wanted:                  0 exclusive:                 11
    current cpu  :                  0 last held:                  1
    current lwp  : 0xffffedeb849ff040 last held: 0xffffedeb7dfdb240
    last locked* : 0xffffffff806d8335 unlocked : 0xffffffff806d8385
    [ 79103.0868574] owner field  : 0xffffedeb7dfdb240 wait/spin:                1/0

Fix it by passing a printer function to lo_dump functions, i.e., make the
functions use db_printf on ddb.

Add missing "static" declaration

Don't check pserialize_not_in_read_section after panic

It can cause a false positive assertion failure.

Fix two oddities...

- remove extraneous semi-colons from do { } while (0) macro definitions
- actually wrap MUTEX_INITIALIZE_ADAPTIVE contents in do { }

Spotted by kre@

Sync with HEAD

Resolve a couple of conflicts (result of the uimin/uimax changes)

Sprinkle __predict_false to dedicated assertions of mutex and rwlock

Check pserialize_not_in_read_section for adaptive mutexes and rwlocks

The overhead of the checks is not negligible so they're turned on only if both
DEBUG and LOCKDEBUG are enabled.

Pull up following revision(s) (requested by ozaki-r in ticket #687):
	sys/kern/kern_rwlock_obj.c: revision 1.4
	sys/rump/librump/rumpkern/locks.c: revision 1.80
	sys/kern/kern_rwlock.c: revision 1.50
	sys/arch/x86/x86/db_memrw.c: revision 1.5,1.6
	sys/ddb/db_command.c: revision 1.150-1.153
	share/man/man4/ddb.4: revision 1.175 (via patch),1.176-1.178
	sys/kern/kern_mutex_obj.c: revision 1.6
	sys/kern/subr_lockdebug.c: revision 1.61-1.64
	sys/sys/lockdebug.h: revision 1.17
	sys/kern/kern_mutex.c: revision 1.71
	sys/sys/lockdebug.h: revision 1.18,1.19
	sys/kern/subr_xcall.c: revision 1.26

Obtain proper initialized addresses of locks allocated by mutex_obj_alloc or rw_obj_alloc

Initialized addresses of locks allocated by mutex_obj_alloc or rw_obj_alloc
were not useful because the addresses were mutex_obj_alloc or rw_obj_alloc
itself. What we want to know are callers of them.

Spinkle ASSERT_SLEEPABLE to xcall functions

Use db_printf instead of printf in ddb

Add a new command, show lockstat, which shows statistics of locks
Currently the command shows the number of allocated locks.
The command is useful only if LOCKDEBUG is enabled.

Add a new command, show all locks, which shows information of active locks

The command shows information of all active (i.e., being held) locks that are
tracked through either of LWPs or CPUs by the LOCKDEBUG facility.  The /t
modifier additionally shows a backtrace for each LWP additionally.  This
feature is useful for debugging especially to analyze deadlocks.
The command is useful only if LOCKDEBUG is enabled.

Don't pass a unset address to lockdebug_lock_print

x86: avoid accessing invalid addresses in ddb like arm32
This avoids that a command stops in the middle of an execution if
a fault occurs due to an access to an invalid address.

Get rid of a redundant output

Improve wording. Fix a Cm argument.

ddb: rename "show lockstat" to "show lockstats" to avoid conflicting with lockstat(8)
Requested by mrg@

add defines to control whether or not mutex operations are skipped
after we have panic'd.  no functional change.

Check if adaptive mutex isn't called in interrupt context

By this check, we can notice this programming error without LOCKDEBUG.
The same check exists in rw_enter.

Obtain proper initialized addresses of locks allocated by mutex_obj_alloc or rw_obj_alloc

Initialized addresses of locks allocated by mutex_obj_alloc or rw_obj_alloc
were not useful because the addresses were mutex_obj_alloc or rw_obj_alloc
itself. What we want to know are callers of them.

Apply C99-style struct initialization to syncobj_t

typo in comment

Pull up following revision(s) (requested by ozaki-r in ticket #495):
	lib/librumpuser/rumpfiber.c: revision 1.13
	lib/librumpuser/rumpuser_pth.c: revision 1.46
	lib/librumpuser/rumpuser_pth_dummy.c: revision 1.18
	sys/kern/kern_condvar.c: revision 1.40
	sys/kern/kern_lock.c: revision 1.161
	sys/kern/kern_mutex.c: revision 1.68
	sys/kern/kern_rwlock.c: revision 1.48
	sys/rump/include/rump/rumpuser.h: revision 1.115
	sys/rump/librump/rumpkern/locks.c: revision 1.76-1.79
Apply C99-style struct initialization to lockops_t
--
Tweak LOCKDEBUG macros (NFC)
--
Distinguish spin mutex and adaptive mutex on rump kernels for LOCKDEBUG
Formerly rump kernels treated the two types of mutexes as both adaptive for
LOCKDEBUG for some reasons.
Now we can detect violations of mutex restrictions on rump kernels such as
taking an adaptive mutex with holding a spin mutex as well as normal kernels.
--
rump: check if the mutex is surely owned by the caller in mutex_exit
Unlocking a not-owned mutex wasn't detected well (it could detect if the mutex
is not held by anyone but that's not enough). Let's check it (the check is the
same as normal kernel's mutex).
If LOCKDEBUG is enabled, give the check over LOCKDEBUG because it can provide
better debugging information.

Apply C99-style struct initialization to lockops_t

update from HEAD

more const

add missing const

Sync with HEAD

Sync with HEAD - tag prg-localcount2-base1

Introduce mutex_ownable() to determine if it is possible for the current
process to acquire a mutex.

Introduce mutex_ownable(9) and use it in localcount_release(9).

Commit of same code to head is awaiting discussion on tech-kern.

Sync with HEAD

Sync with HEAD

Sync with HEAD

For LOCKDEBUG:
Always provide the location of the caller of the lock as __func__, __LINE__.

Sync with HEAD

KNF. Remove extra spaces. No functional change.

Sync with HEAD

typo

Consistently use KPREEMPT_*() here.

Allow MD code to use something other than __cpu_simple_lock_t for doing spin
locks but use the same logic.

MUTEX_SPINBIT_LOCK_INIT
MUTEX_SPINBIT_LOCK_TRY
MUTEX_SPINBIT_LOCKED_P
MUTEX_SPINBIT_LOCK_UNLOCK
MUTEX_INITIALIZE_SPIN_IPL

For those platforms without sub-word CAS, you can use these to place the
mutex lock and ipl inside a full machine word and use CAS to update them.

Don't next structure and enum definitions.
Don't use C++ keywords new, try, class, private, etc.

Rebase to HEAD as of a few days ago.

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs.  ("Protocol error: too many arguments")

sync with head

use __USE() where appropriate.

Remove redundant declaration of MUTEX_CLEAR_WAITERS without checking if
any of the !__HAVE_SIMPLE_MUTEX architectures need it based on the
assumption that HPPA is the only member of that category.

Revert previous - it was wrong. hi joerg.

MUTEX_CLEAR_WAITERS is only ever used by !__HAVE_SIMPLE_MUTEXES

resync from head

Revert change that allowed rw_tryenter(&lock, RW_READER) to recurse
for vfs_busy(). This is no longer necessary.

sync with head

sync to latest -current.

{mutex,rw}_vector_enter: use macro versions to disable/enable preemption.

sync to -current.

Remove COMPAT_SA / KERN_SA.  Welcome to 6.99.3!
Approved by core@.

Sync with HEAD.

sync with head

G/C unused MUTEX_COUNT_BIAS (it was for VAX)

Optimise mutex_onproc() and rw_onproc() by making them O(1), instead
of O(ncpu) for adaptive paths.  Add an LWP destructor, lwp_dtor() with
a comment describing the principle of this barrier.

Reviewed by yamt@ and ad@.

- include opt_multiprocessor.h for explicit MULTIPROCESSOR dependency

Sync with HEAD.

sync with head

Re-apply:

	Invert the sense of the bit to mark if LOCKDEBUG is enabled to
	disabled.

	This will help my fellow developers spot "use before initialised"
	problems that hppa picks up very well.

but fix the !LOCKDEBUG case by defining the "no debug" bits to zero so
they have no effect on lock stubs.

Revert commit from Fri Feb  5 06:43:17 UTC 2010 by skrll:

      Invert the sense of the bit to mark if LOCKDEBUG is enabled to disabled.

      This will help my fellow developers spot "use before initialised" problems
      that hppa picks up very well.

It has to be done differently, because the semantics of mtx_owner in the non-
LOCKDEBUG case can vary significantly between archs, and thus it is not
possible to simply flip a bit to 1.

Ok core@, as at least i386 is unbootable right now.

Invert the sense of the bit to mark if LOCKDEBUG is enabled to disabled.

This will help my fellow developers spot "use before initialised" problems
that hppa picks up very well.

Heave-ho mutex/rwlock object routines into separate modules -- they
don't have anything to do with the lock internals.

sync with head.

Sync with HEAD.

mutex_vector_enter: few predictions.

Sync with HEAD.

Sync with HEAD.

Merge wrstuden-revivesa into HEAD.

Disable generating an upcall for blocking on an adaptive mutex. While
we are indeed blocking, there really is no way for the app to take
advantage of this. We could, for instance, be sleeping on p_lock!

Sync w/ -current. 34 merge conflicts to follow.

sync with head

Sync with HEAD.

Use __noinline.

LOCKDEBUG:

- Tweak it so it can also catch common errors with condition variables.
  The change to kern_condvar.c is not included in this commit and will
  come later.

- Don't call kmem_alloc() if operating in interrupt context, just fail
  the allocation and disable debugging for the object. Makes it safe
  to do mutex_init/rw_init/cv_init in interrupt context, when running
  a LOCKDEBUG kernel.

Reduce ifdefs due to MULTIPROCESSOR slightly.

sync with head.

sync with head.

Allow rw_tryenter(&lock, RW_READER) to recurse, for vfs_busy().

mutex_vector_enter: fix a typo in a comment.

Remove clause 3 and 4 from TNF licenses

MUTEX_SPIN_SPLRAISE: add another __insn_barrier() for safety.

Minor fix for preemption safety.

mutex_owned, rw_read_held, rw_write_held, rw_lock_held: check for a NULL
pointer.

mutex_vector_enter: reduce reads of mtx_owner slightly.

Sync with HEAD.

mutex_vector_exit: add another panicstr check.

Remove dead code from previous.

Add code for dynamically allocated mutexes, as posted on tech-kern.

Sync with HEAD.

sync with head

sync with HEAD

Sync with HEAD

simple_lock_only_held() is gone.

sys/lock.h is required now. ok ad

Start detangling lock.h from intr.h. This is likely to cause short term
breakage, but the mess of dependencies has been regularly breaking the
build recently anyhow.

Sync with HEAD

Sync with HEAD.

Sync with head.

Add back mutex_owner() for Solaris compat.

Sync with HEAD.

Sync with HEAD.

Sync with head.

sync with head

mutex_init: use 'if' instead of 'switch' to avoid complaints from the
compiler.

mutex_init: make MUTEX_DEFAULT + IPL_SOFT* return an adaptive mutex.

Sync with HEAD.

Use membar_*().

Sync with HEAD.

Sync with HEAD

make kmutex_t and krwlock_t smaller by killing lock id.
ok'ed by Andrew Doran.

Sync with HEAD.

sync with head.

Sync with HEAD

Sync with HEAD.

sync with -HEAD

Merge from vmlocking:

- pool_cache changes.
- Debugger/procfs locking fixes.
- Other minor changes.

sync with HEAD

Sync with HEAD.

fix typo in comment (greetings to ad, someone read your longwinded
prattling dissertation)

sync with head.

Sync with HEAD.

Follow the merge of pmap.c on i386 and amd64 and move
pmap_init_tmp_pgtbl into arch/x86/x86/pmap.c. Modify the ACPI wakeup
code to restore CR4 before jumping back into kernel space as the large
page option might cover that.

Sync with HEAD.

machine/{bus,cpu,intr}.h -> sys/{bus,cpu,intr}.h

sync with head.

Merge from vmlocking:

- G/C spinlockmgr() and simple_lock debugging.
- Always include the kernel_lock functions, for LKMs.
- Slightly improved subr_lockdebug code.
- Keep sizeof(struct lock) the same if LOCKDEBUG.

Sync with head.

Sync with head.

Sync with HEAD.

Include sys/cdefs.h first.

__FUNCTION__ -> __func__

Merge nick-csl-alignment.

sync with head.

Apply a patch from yamt@: collapse MUTEX_DEFAULT and MUTEX_DRIVER to mean
the same thing. Only enable adaptive mutexes for IPL_SOFTBIO and IPL_NONE
right now.

Track where locks were initialized.

mutex_vector_exit: fix a lock/unlock inversion.

Simplify the code a bit.

Initial work on provided correctly aligned __cpu_simple_lock_t for hppa
and first attempt at adapting i386 to the changes.

More to come.

KASSERT(!cpu_intr_p()) in a couple of places.

Sync with head.

Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

Sync with head.

merge yamt-idlelwp branch.  asked by core@.  some ports still needs work.

from doc/BRANCHES:

	idle lwp, and some changes depending on it.

	1. separate context switching and thread scheduling.
	   (cf. gmcgarry_ctxsw)
	2. implement idle lwp.
	3. clean up related MD/MI interfaces.
	4. make scheduler(s) modular.

- Pass the error number and residual count to biodone(), and let it handle
  setting error indicators. Prepare to eliminate B_ERROR.
- Add a flag argument to brelse() to be set into the buf's flags, instead
  of doing it directly. Typically used to set B_INVAL.
- Add a "struct cpu_info *" argument to kthread_create(), to be used to
  create bound threads. Change "bool mpsafe" to "int flags".
- Allow exit of LWPs in the IDL state when (l != curlwp).
- More locking fixes & conversion to the new API.

Pull up a change from the vmlocking branch:

- Ensure that LWPs going to sleep are on the sleep queue before releasing
  any interlocks. This is so that calls to turnstile_wakeup will have the
  correct locks held when adjusting priority. Avoids another deadlock.
- Assume that LWPs blocked on a turnstile will never be swapped out.
- LWPs blocking on a turnstile must have kernel priority, as they
  are consuming kernel resources.

In the simple case, provide a generic CAS macro if the MD header does
not provide one.

Partially back out 1.11.2.4. It does not work with more than two CPUs,
and blocking is the best way to solve the problem. The issue with
pressure on other locks has been fixed by restoring kernel priority to
objects blocking on turnstiles.

- Ensure that that LWPs going to sleep are on the sleep queue and so
  have their syncobj pointer updated, so that calls to turnstile_wakeup
  will have the correct locks held when adjusting the current LWP's
  priority. Avoids another deadlock.
- Assume that LWPs blocked on a turnstile will never be swapped out.
- LWPs blocking on a turnstile must have kernel priority, as they
  are consuming kernel resources.

- Blocking in order to avoid deadlocking against the kernel_lock turns
  out to be not so good, since it puts a lot of back pressure onto other
  locks. Instead, just drop kernel_lock and re-acquire it when returning
  from mutex_vector_exit. In the LOCKDEBUG case, continue to block as it
  helps to expose locking bugs.
- Allow NODEBUG mutexes to be spin mutexes, also. To be revisited.

sync with head.

GC the simplelock/spinlock debugging stuff.

Pull in the initial set of changes for the vmlocking branch.

Sync with head.

mutex_vector_enter: put a read memory barrier between the final check that
(1) the holder is no longer running and (2) the waiters bit is set. Needed
to ensure that the operations happen in the correct order.

Sync with HEAD.

Define/use MUTEX_COUNT_BIAS for those implementations that need one (like
vax).

Add a MUTEX_NODEBUG type that avoids allocating debugging resources for
the lock. To be used for e.g. struct vm_page until a better way is found.

mutex_onproc: remove unnecessary memory barriers.

change an error message from:
	Mutex error: sched_unlock_idle: sched_unlock_idle
to
	Mutex error: sched_unlock_idle: sched_mutex not locked

Remove extra space so that symbol renaming works properly.

kern_mutex.c:190: macro `MUTEX_RECEIVE' used without args
kern_mutex.c:199: macro `MUTEX_RECEIVE' used without args
kern_mutex.c:206: macro `MUTEX_GIVE' used without args

- sync with head.
- move sched_changepri back to kern_synch.c as it doesn't know PPQ anymore.

- Nuke mutex_owner()
- mutex_getowner() -> mutex_owner()

implement priority inheritance.

sync with head.

update a comment.

- separate context switching and thread scheduling.
- introduce idle lwp.
- change some related MD/MI interfaces and implement i386 version.

- Add a mutex_wakeup() which clears the waiters flag and wakes all waiters.
  Will be needed to fix a problem with mutexes on sparc.
- mutex_tryenter(): fix a false "locking against myself" error.

- Add/correct comments.
- Print correct function name when an assertion triggers.

Merge newlock2 to head.

- Change syncobj_t::sobj_changepri() to alter both the user priority and
  the effective priority of LWPs. How the effective priority is adjusted
  depends on the type of object.
- Add a couple of comments to sched_kpri() and remrunqueue().

lockstat:

- Cache enabled/disabled status on entry.
- Don't read the cycle counter unless enabled.

Don't signal the lockdebug code that we want to acquire a spin mutex
before raising the SPL, otherwise a spurious 'locking against myself'
error could be triggered.

Play nicer when the system is panicing/cold.

Handle hppa weirdness.

Catch deadlocks earlier.

For now always call sched_unlock_idle/sched_lock_idle. They will be
removed by yamt's cpu_switchto() changes.

- Fix sequence error between saving/raising the SPL.
- Changes for JavaStation.
- Fix bugs with mips & sparc support routines.

Rename some functions to better describe what they do.

Fix detection of deadlock against the big lock.

Minor corrections.

Fix 'SPL NOT LOWERED ON SYSCALL EXIT' with amd64.

Sync with head.

Checkpoint work in progress.

Checkpoint work in progress.

Sync with head.

Checkpoint work in progress.

Compile fixes.

- Update for lockdebug changes.
- Define mutex_getspl(), mutex_setspl() for modifying saved SPL value.

Don't do the simplelock check if panicstr is set.

Add updated locking primatives.

Wrap the stuff that manipulates the guts of a mutex in __MUTEX_PRIVATE.

Cast the return value of __builtin_return_address().

Add some minimal mutex debugging infrastructure.
* struct mutex_debug_info, which MD code adds to the kmutex_t
  when MUTEX_DEBUG is defined, contains the PC values of the
  last-locked and last-unlocked.
* MD code is expected to set last-locked in the enter fast path.
* MD code is expected to always call mutex_vector_exit() in the
  exit case.

Note: this code relies on MD code performing a tail-call to the
slow-path.

Add a "nextproc" argument to turnstile_wakeup() to selectively
wake up a single thread.  This will allow us to implement direct
handoff of rwlocks.

Make turnstiles actually have two queues (reader and writer).  Users
of turnsitles still don't differentiate between them.

mutex_vector_exit(): be more informative if we panic.

Implement mutex_owned() (returns true if current thread owns mutex).

Implement mutex_owner().

New kernel mutex implementation, modeled after those found in Solaris
and described in:

	Solaris Internals: Core Kernel Architecture, Jim Mauro and
	    Richard McDougall, section 3.5.

Both adaptive (spin/sleep) and spin (spin + block interrupts) mutexes
are implemented.  See <sys/mutex.h> for the support that machine-dependent
code must supply.

file kern_mutex.c was initially added on branch newlock.