CVS log for src/sys/kern/kern_rwlock.c

Up to [cvs.NetBSD.org] / src / sys / kern

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

kern_rwlock.c: Sort includes.  No functional change intended.

sys/lwp.h: Nix sys/syncobj.h dependency.

Remove it in ddb/db_syncobj.h too.

New sys/wchan.h defines wchan_t so that users need not pull in
sys/syncobj.h to get it.

Sprinkle #include <sys/syncobj.h> in .c files where it is now needed.

Turnstiles: use the syncobj name for ps/top wmesg when sleeping since it's
more informative than "tstile".

- Simplify how priority boost for blocking in kernel is handled.  Rather
  than setting it up at each site where we block, make it a property of
  syncobj_t.  Then, do not hang onto the priority boost until userret(),
  drop it as soon as the LWP is out of the run queue and onto a CPU.
  Holding onto it longer is of questionable benefit.

- This allows two members of lwp_t to be deleted, and mi_userret() to be
  simplified a lot (next step: trim it down to a single conditional).

- While here, constify syncobj_t and de-inline a bunch of small functions
  like lwp_lock() which turn out not to be small after all (I don't know
  why, but atomic_*_relaxed() seem to provoke a compiler shitfit above and
  beyond what volatile does).

Remove dodgy and unused mutex_owner_running() & rw_owner_running().

Pull up following revision(s) (requested by riastradh in ticket #1861):

	sys/kern/kern_rwlock.c: revision 1.68 (patch)

rwlock(9): Fix membars.

rw_downgrade must be a release operation, and rw_tryupgrade must be
an acquire operation.  membar_producer is not enough -- need to use
membar_release and membar_acquire.

Discussed on tech-kern:
https://mail-index.netbsd.org/tech-kern/2023/02/22/msg028726.html

Pull up following revision(s) (requested by riastradh in ticket #1678):

	sys/kern/kern_rwlock.c: revision 1.68 (patch)

rwlock(9): Fix membars.

rw_downgrade must be a release operation, and rw_tryupgrade must be
an acquire operation.  membar_producer is not enough -- need to use
membar_release and membar_acquire.

Discussed on tech-kern:
https://mail-index.netbsd.org/tech-kern/2023/02/22/msg028726.html

Pull up following revision(s) (requested by riastradh in ticket #266):

	sys/kern/kern_rwlock.c: revision 1.68

rwlock(9): Fix membars.

rw_downgrade must be a release operation, and rw_tryupgrade must be
an acquire operation.  membar_producer is not enough -- need to use
membar_release and membar_acquire.

Discussed on tech-kern:
https://mail-index.netbsd.org/tech-kern/2023/02/22/msg028726.html

Pull up following revision(s) (requested by riastradh in ticket #1860):

	sys/kern/kern_rwlock.c: revision 1.67
	sys/kern/kern_lock.c: revision 1.182
	sys/kern/kern_mutex.c: revision 1.102
	(all via patch)

Sprinkle __predict_{true,false} for panicstr checks

Pull up following revision(s) (requested by riastradh in ticket #1677):

	sys/kern/kern_rwlock.c: revision 1.67
	sys/kern/kern_lock.c: revision 1.182
	sys/kern/kern_mutex.c: revision 1.102

Sprinkle __predict_{true,false} for panicstr checks

Pull up following revision(s) (requested by riastradh in ticket #265):

	sys/kern/kern_rwlock.c: revision 1.67
	sys/kern/kern_lock.c: revision 1.182
	sys/kern/kern_mutex.c: revision 1.102

Sprinkle __predict_{true,false} for panicstr checks

kern: New struct syncobj::sobj_name member for diagnostics.

XXX potential kernel ABI change -- not sure any modules actually use
struct syncobj but it's hard to rule that out because sys/syncobj.h
leaks into sys/lwp.h

rwlock(9): Omit needless macro indirection for membars.

No functional change intended.

kern: Eliminate most __HAVE_ATOMIC_AS_MEMBAR conditionals.

I'm leaving in the conditional around the legacy membar_enters
(store-before-load, store-before-store) in kern_mutex.c and in
kern_lock.c because they may still matter: store-before-load barriers
tend to be the most expensive kind, so eliding them is probably
worthwhile on x86.  (It also may not matter; I just don't care to do
measurements right now, and it's a single valid and potentially
justifiable use case in the whole tree.)

However, membar_release/acquire can be mere instruction barriers on
all TSO platforms including x86, so there's no need to go out of our
way with a bad API to conditionalize them.  If the procedure call
overhead is measurable we just could change them to be macros on x86
that expand into __insn_barrier.

Discussed on tech-kern:
https://mail-index.netbsd.org/tech-kern/2023/02/23/msg028729.html

rwlock(9): Fix membars.

rw_downgrade must be a release operation, and rw_tryupgrade must be
an acquire operation.  membar_producer is not enough -- need to use
membar_release and membar_acquire.

Discussed on tech-kern:
https://mail-index.netbsd.org/tech-kern/2023/02/22/msg028726.html

XXX pullup-8
XXX pullup-9
XXX pullup-10

Sprinkle __predict_{true,false} for panicstr checks

rwlock(9): Convert to membar_acquire/release.

Leave an XXX comment where I suspect there might be a missing membar
-- to be audited when I have more time to think!

Merge changes from current as of 20200406

Sync with head.

Add rw_lock_op(): return either RW_READER or RW_WRITER for a lock that is
known to be held by the caller.  Panic if the lock is not held.

Sync with head.

Remove unintentional differences to base.

Add a comment.

Back out previous.

Add a rw_owner_running() for the pagedaemon.

PR kern/54880: -current hangs in mountroot

- Don't set the RW_NODEBUG flag on init, since assembly stubs can't handle it.
- rw_downgrade(): fix a case where the RW_NODEBUG flag was lost.

Adaptive rwlocks proposed on tech-kern and working well on this branch
with vnode locks.

Sync with head.

Tidy rwlocks a bit, no functional change intended.  Mainly:

- rw_downgrade(): do it in a for () loop like all the others.
- Explicitly carry around RW_NODEBUG - don't be lazy.
- Remove pointless macros.
- Don't make assertions conditional on LOCKDEBUG, there's no reason.
- Make space for a new flag bit (not added yet).

Sync with head.

- Shuffle some items around in struct lwp to save space.  Remove an unused
  item or two.

- For lockstat, get a useful callsite for vnode locks (caller to vn_lock()).

Expunge the panicstr checks, we don't need them.

Back out previous.  It works on amd64 under stress test but not
evbarm-aarch64 for some reason.  Will revisit.

A couple more tweaks to avoid reading the lock word.

Largely eliminate the MD rwlock.h header file.

This was full of definitions that have been obsolete for over a
decade.  The file still remains for __HAVE_RW_STUBS but that's all.
Used only internally in kern_rwlock.c now, not by <sys/rwlock.h>.

Remove some unneeded memory barriers and reads of the lock word.
Prodded by Mateusz Guzik.

Sync with HEAD

Avoid prepending a timestamp to lock debug outputs on ddb

Lock printer functions (lockops_t#lo_dump) use printf_nolog to print, but
printf_nolog now prepends a timestamp which is unnecessary for ddb:

    db{0}> show all locks/t
    [Locks tracked through LWPs]
    Locks held by an LWP (iperf):
    Lock 0 (initialized at soinit)
    lock address : 0xffffedeb84b06080 type     :     sleep/adaptive
    initialized  : 0xffffffff806d8c3f
    shared holds :                  0 exclusive:                  1
    shares wanted:                  0 exclusive:                 11
    current cpu  :                  0 last held:                  1
    current lwp  : 0xffffedeb849ff040 last held: 0xffffedeb7dfdb240
    last locked* : 0xffffffff806d8335 unlocked : 0xffffffff806d8385
    [ 79103.0868574] owner field  : 0xffffedeb7dfdb240 wait/spin:                1/0

Fix it by passing a printer function to lo_dump functions, i.e., make the
functions use db_printf on ddb.

Don't check pserialize_not_in_read_section after panic

It can cause a false positive assertion failure.

Sync with HEAD

Resolve a couple of conflicts (result of the uimin/uimax changes)

Sprinkle __predict_false to dedicated assertions of mutex and rwlock

Check pserialize_not_in_read_section for adaptive mutexes and rwlocks

The overhead of the checks is not negligible so they're turned on only if both
DEBUG and LOCKDEBUG are enabled.

Pull up following revision(s) (requested by ozaki-r in ticket #687):
	sys/kern/kern_rwlock_obj.c: revision 1.4
	sys/rump/librump/rumpkern/locks.c: revision 1.80
	sys/kern/kern_rwlock.c: revision 1.50
	sys/arch/x86/x86/db_memrw.c: revision 1.5,1.6
	sys/ddb/db_command.c: revision 1.150-1.153
	share/man/man4/ddb.4: revision 1.175 (via patch),1.176-1.178
	sys/kern/kern_mutex_obj.c: revision 1.6
	sys/kern/subr_lockdebug.c: revision 1.61-1.64
	sys/sys/lockdebug.h: revision 1.17
	sys/kern/kern_mutex.c: revision 1.71
	sys/sys/lockdebug.h: revision 1.18,1.19
	sys/kern/subr_xcall.c: revision 1.26

Obtain proper initialized addresses of locks allocated by mutex_obj_alloc or rw_obj_alloc

Initialized addresses of locks allocated by mutex_obj_alloc or rw_obj_alloc
were not useful because the addresses were mutex_obj_alloc or rw_obj_alloc
itself. What we want to know are callers of them.

Spinkle ASSERT_SLEEPABLE to xcall functions

Use db_printf instead of printf in ddb

Add a new command, show lockstat, which shows statistics of locks
Currently the command shows the number of allocated locks.
The command is useful only if LOCKDEBUG is enabled.

Add a new command, show all locks, which shows information of active locks

The command shows information of all active (i.e., being held) locks that are
tracked through either of LWPs or CPUs by the LOCKDEBUG facility.  The /t
modifier additionally shows a backtrace for each LWP additionally.  This
feature is useful for debugging especially to analyze deadlocks.
The command is useful only if LOCKDEBUG is enabled.

Don't pass a unset address to lockdebug_lock_print

x86: avoid accessing invalid addresses in ddb like arm32
This avoids that a command stops in the middle of an execution if
a fault occurs due to an access to an invalid address.

Get rid of a redundant output

Improve wording. Fix a Cm argument.

ddb: rename "show lockstat" to "show lockstats" to avoid conflicting with lockstat(8)
Requested by mrg@

Obtain proper initialized addresses of locks allocated by mutex_obj_alloc or rw_obj_alloc

Initialized addresses of locks allocated by mutex_obj_alloc or rw_obj_alloc
were not useful because the addresses were mutex_obj_alloc or rw_obj_alloc
itself. What we want to know are callers of them.

Apply C99-style struct initialization to syncobj_t

Pull up following revision(s) (requested by ozaki-r in ticket #495):
	lib/librumpuser/rumpfiber.c: revision 1.13
	lib/librumpuser/rumpuser_pth.c: revision 1.46
	lib/librumpuser/rumpuser_pth_dummy.c: revision 1.18
	sys/kern/kern_condvar.c: revision 1.40
	sys/kern/kern_lock.c: revision 1.161
	sys/kern/kern_mutex.c: revision 1.68
	sys/kern/kern_rwlock.c: revision 1.48
	sys/rump/include/rump/rumpuser.h: revision 1.115
	sys/rump/librump/rumpkern/locks.c: revision 1.76-1.79
Apply C99-style struct initialization to lockops_t
--
Tweak LOCKDEBUG macros (NFC)
--
Distinguish spin mutex and adaptive mutex on rump kernels for LOCKDEBUG
Formerly rump kernels treated the two types of mutexes as both adaptive for
LOCKDEBUG for some reasons.
Now we can detect violations of mutex restrictions on rump kernels such as
taking an adaptive mutex with holding a spin mutex as well as normal kernels.
--
rump: check if the mutex is surely owned by the caller in mutex_exit
Unlocking a not-owned mutex wasn't detected well (it could detect if the mutex
is not held by anyone but that's not enough). Let's check it (the check is the
same as normal kernel's mutex).
If LOCKDEBUG is enabled, give the check over LOCKDEBUG because it can provide
better debugging information.

Apply C99-style struct initialization to lockops_t

update from HEAD

more const

Sync with HEAD

Sync with HEAD

Sync with HEAD

For LOCKDEBUG:
Always provide the location of the caller of the lock as __func__, __LINE__.

Consistently use KPREEMPT_*() here.

Don't next structure and enum definitions.
Don't use C++ keywords new, try, class, private, etc.

Rebase to HEAD as of a few days ago.

Rebase.

Fix typo in comment

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs.  ("Protocol error: too many arguments")

sync with head

use __USE() where appropriate.

Typo in comment.

resync from head

Revert change that allowed rw_tryenter(&lock, RW_READER) to recurse
for vfs_busy(). This is no longer necessary.

resync with head

sync with (a bit old) head

rw_vector_enter: reload owner in the case of no hand-off.
this fixes crashes in rw_oncpu().

sync with head

sync to latest -current.

{mutex,rw}_vector_enter: use macro versions to disable/enable preemption.

Sync with HEAD.

sync with head

Optimise mutex_onproc() and rw_onproc() by making them O(1), instead
of O(ncpu) for adaptive paths.  Add an LWP destructor, lwp_dtor() with
a comment describing the principle of this barrier.

Reviewed by yamt@ and ad@.

- include opt_multiprocessor.h for explicit MULTIPROCESSOR dependency

Sync with HEAD.

sync to netbsd-5

sync with head

Re-apply:

	Invert the sense of the bit to mark if LOCKDEBUG is enabled to
	disabled.

	This will help my fellow developers spot "use before initialised"
	problems that hppa picks up very well.

but fix the !LOCKDEBUG case by defining the "no debug" bits to zero so
they have no effect on lock stubs.

Revert commit from Fri Feb  5 06:43:17 UTC 2010 by skrll:

      Invert the sense of the bit to mark if LOCKDEBUG is enabled to disabled.

      This will help my fellow developers spot "use before initialised" problems
      that hppa picks up very well.

It has to be done differently, because the semantics of mtx_owner in the non-
LOCKDEBUG case can vary significantly between archs, and thus it is not
possible to simply flip a bit to 1.

Ok core@, as at least i386 is unbootable right now.

Invert the sense of the bit to mark if LOCKDEBUG is enabled to disabled.

This will help my fellow developers spot "use before initialised" problems
that hppa picks up very well.

Heave-ho mutex/rwlock object routines into separate modules -- they
don't have anything to do with the lock internals.

Sync with HEAD.

Pull up following revision(s) (requested by rmind in ticket #837):
	sys/kern/kern_rwlock.c: revision 1.31
lockdebug fixes for rw_tryupgrade/rw_downgrade.

Pull up following revision(s) (requested by rmind in ticket #837):
	sys/kern/kern_rwlock.c: revision 1.31
lockdebug fixes for rw_tryupgrade/rw_downgrade.

sync with head

rw_vector_exit: remove a redundant condition.

Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.

Pull up following revision(s) (requested by ad in ticket #725):
	sys/kern/kern_rwlock.c: revision 1.30
A workaround for a bug with some Opteron revisions where locked operations
sometimes do not serve as memory barriers, allowing memory references to
bleed outside of critical sections.  It is possible that this is the
reason for pkgbuild's longstanding crashiness.
For rwlocks, always enable the explicit membars. They were disabled only
on x86, and since they are not in the fast-path it's not a big deal.
TODO: convert these to an atomic_membar_foo() or similar that does ordering
between regular data references and atomic references.

Pull up following revision(s) (requested by ad in ticket #725):
	sys/kern/kern_rwlock.c: revision 1.30
A workaround for a bug with some Opteron revisions where locked operations
sometimes do not serve as memory barriers, allowing memory references to
bleed outside of critical sections.  It's possible that this is the
reason for pkgbuild's longstanding crashiness.
For rwlocks, always enable the explicit membars. They were disabled only
on x86, and since they are not in the fast-path it's not a big deal.
TODO: convert these to an atomic_membar_foo() or similar that does ordering
between regular data references and atomic references.

lockdebug fixes for rw_tryupgrade/rw_downgrade.

sync with head.

Sync with HEAD.

A workaround for a bug with some Opteron revisions where locked operations
sometimes do not serve as memory barriers, allowing memory references to
bleed outside of critical sections.  It's possible that this is the
reason for pkgbuild's longstanding crashiness.

For rwlocks, always enable the explicit membars. They were disabled only
on x86, and since they are not in the fast-path it's not a big deal.
TODO: convert these to an atomic_membar_foo() or similar that does ordering
between regular data references and atomic references.

Add rw_obj_*() functions to mirror the existing mutex functions.
Proposed on tech-kern quite some time ago.

Sync with HEAD.

Sync with HEAD.

Sync with wrstuden-revivesa-base-2.

Sync with head.

rw_vector_exit(): When doing direct hand-off to a writer, only set
RW_WRITE_WANTED if there are additional writers waiting.

Sync with head.

PR/39155: Wolfgang Solfrank: Invalid parameter to turnstile_exit in rw_downgrade.

Sync w/ -current. 34 merge conflicts to follow.

sync with head

Sync with HEAD.

Use __noinline.

LOCKDEBUG:

- Tweak it so it can also catch common errors with condition variables.
  The change to kern_condvar.c is not included in this commit and will
  come later.

- Don't call kmem_alloc() if operating in interrupt context, just fail
  the allocation and disable debugging for the object. Makes it safe
  to do mutex_init/rw_init/cv_init in interrupt context, when running
  a LOCKDEBUG kernel.

Reduce ifdefs due to MULTIPROCESSOR slightly.

sync with head.

sync with head.

Allow rw_tryenter(&lock, RW_READER) to recurse, for vfs_busy().

Remove clause 3 and 4 from TNF licenses

mutex_owned, rw_read_held, rw_write_held, rw_lock_held: check for a NULL
pointer.

rwlock changes, discussed on tech-kern:

- Use atomic ops directly, since rwlocks work the same way on all platforms.
- Try to make it a bit more cache efficient, and use branch hints.
- Fix a bug in rw_downgrade() where the turnstile lock was not released.
- Remove a couple of redundant assertions.
- Use atomic_swap instead of atomic_cas where it's safe to do so.
- After acquiring the turnstile lock in rw_vector_enter, check if the
  owner is running again and spin if so.
- Introduce and use rw_onproc() instead of abusing mutex_onproc().
- Change the handoff/release algorithm to reduce the window when a rwlock
  can held, but the owner not on a CPU.

Do adaptive spinning for rwlocks, but only if the lock is write held and
there are no waiters. This gives a major boost to build.sh on larger
systems as directory vnode locks are exclusive for lookup, but are often
only held for a very short period of time.

This change has the potential to more readily expose lock order reversals
and other types of deadlock.

sync with HEAD

Sync with HEAD.

sync with head.

Fix error in previous. Spotted by yamt@.

rw_write_held: return true only if held by the current LWP.

- Prevent LOCKDEBUG from complaining if we do rw_tryenter() on a lock
  that we already own.
- Add an assembly stub for rw_tryenter().

sync with head

sync with HEAD

Sync with HEAD

Need sys/lock.h for kernel_lock.

Pull in sys/atomic.h for memory barriers.

Sync with HEAD.

Sync with HEAD.

Sync with head.

sync with head

rw_vector_enter: assert !cpu_intr_p().

Sync with HEAD.

Sync with HEAD

make kmutex_t and krwlock_t smaller by killing lock id.
ok'ed by Andrew Doran.

sync with HEAD

sync with head.

Sync with HEAD.

Follow the merge of pmap.c on i386 and amd64 and move
pmap_init_tmp_pgtbl into arch/x86/x86/pmap.c. Modify the ACPI wakeup
code to restore CR4 before jumping back into kernel space as the large
page option might cover that.

sync with head.

Merge from vmlocking:

- G/C spinlockmgr() and simple_lock debugging.
- Always include the kernel_lock functions, for LKMs.
- Slightly improved subr_lockdebug code.
- Keep sizeof(struct lock) the same if LOCKDEBUG.

Sync with head.

Sync with HEAD.

Include sys/cdefs.h first.

__FUNCTION__ -> __func__

sync with head.

Track where locks were initialized.

Simplify the code a bit.

KASSERT(!cpu_intr_p()) in a couple of places.

Sync with head.

merge yamt-idlelwp branch.  asked by core@.  some ports still needs work.

from doc/BRANCHES:

	idle lwp, and some changes depending on it.

	1. separate context switching and thread scheduling.
	   (cf. gmcgarry_ctxsw)
	2. implement idle lwp.
	3. clean up related MD/MI interfaces.
	4. make scheduler(s) modular.

Pull up a change from the vmlocking branch:

- Ensure that LWPs going to sleep are on the sleep queue before releasing
  any interlocks. This is so that calls to turnstile_wakeup will have the
  correct locks held when adjusting priority. Avoids another deadlock.
- Assume that LWPs blocked on a turnstile will never be swapped out.
- LWPs blocking on a turnstile must have kernel priority, as they
  are consuming kernel resources.

In the simple case, provide a generic CAS macro if the MD header does
not provide one.

sync with head.

- Ensure that that LWPs going to sleep are on the sleep queue and so
  have their syncobj pointer updated, so that calls to turnstile_wakeup
  will have the correct locks held when adjusting the current LWP's
  priority. Avoids another deadlock.
- Assume that LWPs blocked on a turnstile will never be swapped out.
- LWPs blocking on a turnstile must have kernel priority, as they
  are consuming kernel resources.

Sync with head.

Cosmetic changes.

GC the simplelock/spinlock debugging stuff.

Sync with HEAD.

Remove extra space so that symbol renaming works properly.

- sync with head.
- move sched_changepri back to kern_synch.c as it doesn't know PPQ anymore.

Add a comment.

implement priority inheritance.

sync with head.

- Add/correct comments.
- Print correct function name when an assertion triggers.

Merge newlock2 to head.

- Change syncobj_t::sobj_changepri() to alter both the user priority and
  the effective priority of LWPs. How the effective priority is adjusted
  depends on the type of object.
- Add a couple of comments to sched_kpri() and remrunqueue().

lockstat:

- Cache enabled/disabled status on entry.
- Don't read the cycle counter unless enabled.

- Decouple downgrade from rw_vector_exit(), and simplify a bit.
- Fix a problem that could cause stalls under load.

Catch deadlocks earlier.

Checkpoint work in progress.

Checkpoint work in progress.

Checkpoint work in progress.

- Update for lockdebug changes.
- Define mutex_getspl(), mutex_setspl() for modifying saved SPL value.

Don't do the simplelock check if panicstr is set.

Add updated locking primatives.

Add some basic debugging infrastructure as found in mutexes.

Change RWLOCK_ACQUIRE() and RWLOCK_RELEASE() to evaluate to true
or false depending on the success of the "compare and swap" operation.

This makes them a lot easier to implement on IA32.

Add a "nextproc" argument to turnstile_wakeup() to selectively
wake up a single thread.  This will allow us to implement direct
handoff of rwlocks.

Make turnstiles actually have two queues (reader and writer).  Users
of turnsitles still don't differentiate between them.

Reader/writer lock implementation, modeled after the Solaris rwlocks,
a description of which can be found in:

	Solaris Internals: Core Kernel Architecture, Jim Mauro and
	    Richard McDougall, section 3.6.

XXX Does not currently distinguish between waiting readers and
    waiting writers.
XXX Because of above, does not yet implement direct-handoff.

file kern_rwlock.c was initially added on branch newlock.