CVS log for src/sys/kern/subr_workqueue.c

Up to [cvs.NetBSD.org] / src / sys / kern

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

Pull up following revision(s) (requested by rin in ticket #821):

	sys/arch/x86/x86/intr.c: revision 1.169
	sys/kern/kern_softint.c: revision 1.76
	sys/kern/subr_workqueue.c: revision 1.48
	sys/kern/kern_idle.c: revision 1.36
	sys/kern/subr_xcall.c: revision 1.38

check that l_nopreempt (preemption count) doesn't change after callbacks

check that the idle loop, soft interrupt handlers, workqueue, and xcall
callbacks do not modify the preemption count, in most cases, knowing it
should be 0 currently.

this work was originally done by simonb.  cleaned up slightly and some
minor enhancement made by myself, and with discussion with riastradh@.
other callback call sites could check this as well (such as MD interrupt
handlers, or really anything that includes a callback registration.  x86
version to be commited separately.)

apply some more diagnostic checks for x86 interrupts
convert intr_biglock_wrapper() into a slight less complete
intr_wrapper(), and move the kernel lock/unlock points into
the new intr_biglock_wrapper().
add curlwp->l_nopreempt checking for interrupt handlers,
including the dtrace wrapper.

XXX: has to copy the i8254_clockintr hack.

tested for a few months by myself, and recently by rin@ on both
current and netbsd-10. thanks!

Pull up following revision(s) (requested by riastradh in ticket #1830):

	sys/kern/subr_workqueue.c: revision 1.40
	sys/kern/subr_workqueue.c: revision 1.41
	sys/kern/subr_workqueue.c: revision 1.42
	sys/kern/subr_workqueue.c: revision 1.43
	sys/kern/subr_workqueue.c: revision 1.44
	sys/kern/subr_workqueue.c: revision 1.45
	sys/kern/subr_workqueue.c: revision 1.46
	tests/rump/kernspace/workqueue.c: revision 1.7
	sys/kern/subr_workqueue.c: revision 1.47
	tests/rump/kernspace/workqueue.c: revision 1.8
	tests/rump/kernspace/workqueue.c: revision 1.9
	tests/rump/rumpkern/t_workqueue.c: revision 1.3
	tests/rump/rumpkern/t_workqueue.c: revision 1.4
	tests/rump/kernspace/kernspace.h: revision 1.9
	tests/rump/rumpkern/Makefile: revision 1.20
	sys/kern/subr_workqueue.c: revision 1.39
	share/man/man9/workqueue.9: revision 1.15
	(all via patch)

workqueue: Lift unnecessary restriction on workqueue_wait.

Allow multiple concurrent waits at a time, and allow enqueueing work
at the same time (as long as it's not the work we're waiting for).

This way multiple users can use a shared global workqueue and safely
wait for individual work items concurrently, while the workqueue is
still in use for other items (e.g., wg(4) peers).

This has the side effect of taking away a diagnostic measure, but I
think allowing the diagnostic's false positives instead of rejecting
them is worth it.  We could cheaply add it back with some false
negatives if it's important.
workqueue(9): workqueue_wait and workqueue_destroy may sleep.

But might not, so assert sleepable up front.
workqueue(9): Sprinkle dtrace probes.
tests/rump/rumpkern: Use PROGDPLIBS, not explicit -L/-l.

This way we relink the t_* test programs whenever changes under
tests/rump/kernspace change libkernspace.a.

workqueue(9) tests: Nix trailing whitespace.

workqueue(9) tests: Destroy struct work immediately on entry.

workqueue(9) tests: Add test for PR kern/57574.

workqueue(9): Avoid touching running work items in workqueue_wait.

As soon as the workqueue function has called, it is forbidden to
touch the struct work passed to it -- the function might free or
reuse the data structure it is embedded in.

So workqueue_wait is forbidden to search the queue for the batch of
running work items.  Instead, use a generation number which is odd
while the thread is processing a batch of work and even when not.
There's still a small optimization available with the struct work
pointer to wait for: if we find the work item in one of the per-CPU
_pending_ queues, then after we wait for a batch of work to complete
on that CPU, we don't need to wait for work on any other CPUs.
PR kern/57574

workqueue(9): Sprinkle dtrace probes for workqueue_wait edge cases.

Let's make it easy to find out whether these are hit.

workqueue(9): Stop violating queue(3) internals.

workqueue(9): Avoid unnecessary mutex_exit/enter cycle each loop.

workqueue(9): Sort includes.
No functional change intended.

workqueue(9): Factor out wq->wq_flags & WQ_FPU in workqueue_worker.
No functional change intended.  Makes it clearer that s is
initialized when used.

check that l_nopreempt (preemption count) doesn't change after callbacks

check that the idle loop, soft interrupt handlers, workqueue, and xcall
callbacks do not modify the preemption count, in most cases, knowing it
should be 0 currently.

this work was originally done by simonb.  cleaned up slightly and some
minor enhancement made by myself, and with discussion with riastradh@.

other callback call sites could check this as well (such as MD interrupt
handlers, or really anything that includes a callback registration.  x86
version to be commited separately.)

Pull up following revision(s) (requested by riastradh in ticket #342):

	sys/kern/subr_workqueue.c: revision 1.42
	sys/kern/subr_workqueue.c: revision 1.43
	sys/kern/subr_workqueue.c: revision 1.44
	sys/kern/subr_workqueue.c: revision 1.45
	sys/kern/subr_workqueue.c: revision 1.46
	tests/rump/kernspace/workqueue.c: revision 1.7
	sys/kern/subr_workqueue.c: revision 1.47
	tests/rump/kernspace/workqueue.c: revision 1.8
	tests/rump/kernspace/workqueue.c: revision 1.9
	tests/rump/rumpkern/t_workqueue.c: revision 1.3
	tests/rump/rumpkern/t_workqueue.c: revision 1.4
	tests/rump/kernspace/kernspace.h: revision 1.9
	tests/rump/rumpkern/Makefile: revision 1.20

tests/rump/rumpkern: Use PROGDPLIBS, not explicit -L/-l.

This way we relink the t_* test programs whenever changes under
tests/rump/kernspace change libkernspace.a.

workqueue(9) tests: Nix trailing whitespace.

workqueue(9) tests: Destroy struct work immediately on entry.

workqueue(9) tests: Add test for PR kern/57574.

workqueue(9): Avoid touching running work items in workqueue_wait.

As soon as the workqueue function has called, it is forbidden to
touch the struct work passed to it -- the function might free or
reuse the data structure it is embedded in.

So workqueue_wait is forbidden to search the queue for the batch of
running work items.  Instead, use a generation number which is odd
while the thread is processing a batch of work and even when not.

There's still a small optimization available with the struct work
pointer to wait for: if we find the work item in one of the per-CPU
_pending_ queues, then after we wait for a batch of work to complete
on that CPU, we don't need to wait for work on any other CPUs.
PR kern/57574

workqueue(9): Sprinkle dtrace probes for workqueue_wait edge cases.

Let's make it easy to find out whether these are hit.

workqueue(9): Stop violating queue(3) internals.

workqueue(9): Avoid unnecessary mutex_exit/enter cycle each loop.

workqueue(9): Sort includes.
No functional change intended.

workqueue(9): Factor out wq->wq_flags & WQ_FPU in workqueue_worker.
No functional change intended.  Makes it clearer that s is
initialized when used.

workqueue(9): Factor out wq->wq_flags & WQ_FPU in workqueue_worker.

No functional change intended.  Makes it clearer that s is
initialized when used.

workqueue(9): Sort includes.

No functional change intended.

workqueue(9): Avoid unnecessary mutex_exit/enter cycle each loop.

workqueue(9): Stop violating queue(3) internals.

workqueue(9): Sprinkle dtrace probes for workqueue_wait edge cases.

Let's make it easy to find out whether these are hit.

workqueue(9): Avoid touching running work items in workqueue_wait.

As soon as the workqueue function has called, it is forbidden to
touch the struct work passed to it -- the function might free or
reuse the data structure it is embedded in.

So workqueue_wait is forbidden to search the queue for the batch of
running work items.  Instead, use a generation number which is odd
while the thread is processing a batch of work and even when not.

There's still a small optimization available with the struct work
pointer to wait for: if we find the work item in one of the per-CPU
_pending_ queues, then after we wait for a batch of work to complete
on that CPU, we don't need to wait for work on any other CPUs.

PR kern/57574

XXX pullup-10
XXX pullup-9
XXX pullup-8

workqueue(9): Sprinkle dtrace probes.

workqueue(9): workqueue_wait and workqueue_destroy may sleep.

But might not, so assert sleepable up front.

workqueue: Lift unnecessary restriction on workqueue_wait.

Allow multiple concurrent waits at a time, and allow enqueueing work
at the same time (as long as it's not the work we're waiting for).
This way multiple users can use a shared global workqueue and safely
wait for individual work items concurrently, while the workqueue is
still in use for other items (e.g., wg(4) peers).

This has the side effect of taking away a diagnostic measure, but I
think allowing the diagnostic's false positives instead of rejecting
them is worth it.  We could cheaply add it back with some false
negatives if it's important.

New workqueue flag WQ_FPU.

Arranges kthread_fpu_enter/exit around calls to the worker.  Saves
cost over explicit calls to kthread_fpu_enter/exit in the worker by
only doing it once, since there's often a high cost to flushing the
icache and zeroing the fpu registers.

As proposed on tech-kern:
https://mail-index.netbsd.org/tech-kern/2020/06/20/msg026524.html

Sync with HEAD

Pull up following revision(s) (requested by ozaki-r in ticket #879):

	sys/kern/subr_workqueue.c: revision 1.37

Don't wait on workqueue_wait if called from worker itself

Otherwise workqueue_wait never return in such a case.  This treatment
is the same as callout_halt.

Don't wait on workqueue_wait if called from worker itself

Otherwise workqueue_wait never return in such a case.  This treatment
is the same as callout_halt.

Check the length of a passed name to avoid slient truncation

Pull up following revision(s) (requested by ozaki-r in ticket #528):
	sys/net/agr/if_agr.c: revision 1.42
	sys/netinet6/nd6_rtr.c: revision 1.137
	sys/netinet6/nd6_rtr.c: revision 1.138
	sys/net/agr/if_agr.c: revision 1.46
	sys/net/route.c: revision 1.206
	sys/net/if.c: revision 1.419
	sys/net/agr/if_agrether.c: revision 1.10
	sys/netinet6/nd6.c: revision 1.241
	sys/netinet6/nd6.c: revision 1.242
	sys/netinet6/nd6.c: revision 1.243
	sys/netinet6/nd6.c: revision 1.244
	sys/netinet6/nd6.c: revision 1.245
	sys/netipsec/ipsec_input.c: revision 1.52
	sys/netipsec/ipsec_input.c: revision 1.53
	sys/net/agr/if_agrsubr.h: revision 1.5
	sys/kern/subr_workqueue.c: revision 1.35
	sys/netipsec/ipsec.c: revision 1.124
	sys/net/agr/if_agrsubr.c: revision 1.11
	sys/net/agr/if_agrsubr.c: revision 1.12
Simplify; share agr_vlan_add and agr_vlan_del (NFCI)
Fix late NULL-checking (CID 1427782: Null pointer dereferences (REVERSE_INULL))
KNF: replace soft tabs with hard tabs
Add missing NULL-checking for m_pullup (CID 1427770: Null pointer dereferences (NULL_RETURNS))
Add locking.
Revert "Get rid of unnecessary splsoftnet" (v1.133)
It's not always true that softnet_lock is held these places.
See PR kern/52947.
Get rid of unnecessary splsoftnet (redo)
Unless NET_MPSAFE, splsoftnet is still needed for rt_* functions.
Use existing fill_[pd]rlist() functions to calculate size of buffer to
allocate, rather than relying on an arbitrary length passed in from
userland.
Allow copyout() of partial results if the user buffer is too small, to
be consistent with the way sysctl(3) is documented.
Garbage-collect now-unused third parrameter in the fill_[pd]rlist()
functions.
As discussed on IRC.
OK kamil@ and christos@
XXX Needs pull-up to netbsd-8 branch.
Simplify, from christos@
More simplification, this time from ozaki-r@
No need to break after return.
One more from christos@
No need to initialize fill_func
more cleanup (don't allow oldlenp == NULL)
Destroy ifq_lock at the end of if_detach
It still can be used in if_detach.
Prevent rt_free_global.wk from being enqueued to workqueue doubly
Check if a queued work is tried to be enqueued again, which is not allowed

Check if a queued work is tried to be enqueued again, which is not allowed

Pull up following revision(s) (requested by ozaki-r in ticket #497):
	tests/rump/rumpkern/Makefile: revision 1.16
	tests/rump/kernspace/Makefile: revision 1.6
	tests/rump/kernspace/workqueue.c: revision 1.1
	tests/rump/kernspace/workqueue.c: revision 1.2
	tests/rump/kernspace/workqueue.c: revision 1.3
	tests/rump/kernspace/workqueue.c: revision 1.4
	tests/rump/kernspace/workqueue.c: revision 1.5
	tests/rump/kernspace/workqueue.c: revision 1.6
	tests/rump/rumpkern/t_workqueue.c: revision 1.1
	sys/sys/workqueue.h: revision 1.10
	tests/rump/rumpkern/t_workqueue.c: revision 1.2
	tests/rump/kernspace/kernspace.h: revision 1.5
	tests/rump/kernspace/kernspace.h: revision 1.6
	sys/net/if_bridge.c: revision 1.147
	distrib/sets/lists/debug/mi: revision 1.225
	sys/kern/subr_workqueue.c: revision 1.34
	share/man/man9/workqueue.9: revision 1.12
	sys/net/if_spppsubr.c: revision 1.178
	distrib/sets/lists/tests/mi: revision 1.763
Add simple test for workqueue(9)
Add declaration. build fix
sorry, I forgot to commit this file.
Tweak use of cv_timedwait
- Handle its return value
- Specify more appropriate time-out periods (2 ticks is too short)
Fix a race condition on taking the mutex
The workqueue worker can take the mutex before the tester tries to take it after
calling workqueue_enqueue. If it happens, the worker calls cv_broadcast before
the tester calls cv_timedwait and the tester will wait until the cv timed out
Take the mutex before calling workqueue_enqueue so that the tester surely calls
cv_timedwait before the worker calls cv_broadcast.
The fix stabilizes the test, t_workqueue/workqueue1.
Add workqueue_wait that waits for a specific work to finish
The caller must ensure that no new work is enqueued before calling
workqueue_wait. Note that Note that if the workqueue is WQ_PERCPU, the caller
can enqueue a new work to another queue other than the waiting queue.
Discussed on tech-kern@
Ensure the timer isn't running by using workqueue_wait
Functionalize some routines to add new tests easily (NFC)
Add a test case for workqueue_wait
Fix build

Add workqueue_wait that waits for a specific work to finish

The caller must ensure that no new work is enqueued before calling
workqueue_wait. Note that Note that if the workqueue is WQ_PERCPU, the caller
can enqueue a new work to another queue other than the waiting queue.

Discussed on tech-kern@

Resync to 2012-11-19 00:00:00 UTC

sync with head

If the workqueue is using a prio less than PRI_KERNEL, make sure KTHREAD_TS
is used when creating the kthread.

Turn a workqueue(9) name into an array in the struct workqueue, rather
than a const char *. This avoids keeping a reference to a string
owned by caller (string could be allocated on stack).

These don't need uvm/uvm_extern.h.

sync with head

workqueue_finiqueue: remove unused variable.

Remove uarea swap-out functionality:

- Addresses the issue described in PR/38828.
- Some simplification in threading and sleepq subsystems.
- Eliminates pmap_collect() and, as a side note, allows pmap optimisations.
- Eliminates XS_CTL_DATA_ONSTACK in scsipi code.
- Avoids few scans on LWP list and thus potentially long holds of proc_lock.
- Cuts ~1.5k lines of code.  Reduces amd64 kernel size by ~4k.
- Removes __SWAP_BROKEN cases.

Tested on x86, mips, acorn32 (thanks <mpumford>) and partly tested on
acorn26 (thanks to <bjh21>).

Discussed on <tech-kern>, reviewed by <ad>.

sync with head.

struct lwp -> lwp_t for consistency

Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.

sync with head.

Sync with HEAD.

Pull up following revision(s) (requested by ad in ticket #651):
	sys/kern/subr_workqueue.c: revision 1.27
workqueue_finiqueue: our stack could be swapped out while enqueued to
a worker thread.

workqueue_finiqueue: our stack could be swapped out while enqueued to
a worker thread.

Sync with HEAD.

Sync with HEAD.

Merge in changes between wrstuden-revivesa-base-2 and
wrstuden-revivesa-base-3.

Sync with wrstuden-revivesa-base-2.

Replace intptr_t in few places to uintptr_t.

Sync with head.

Sync with HEAD.

Switch from KASSERT to CTASSERT for those asserts testing sizes of types.

Sync with HEAD.

Replace use of CACHE_LINE_SIZE in some obvious places.

sync with head.

sync with HEAD

sync with head.

Use cpu index instead of the machine dependend, not very expressive
cpuid when naming user-visible kernel entities.

sync with HEAD

Sync with HEAD.

Sync with HEAD.

Sync with head.

sync with head

Match the docs: MUTEX_DRIVER/SPIN are now only for porting code written
for Solaris.

sync with head.

Sync with HEAD.

Sync with HEAD.

Sync with HEAD.

don't bother to set thread's priority by ourselves,
as kthread_create does it for us now.  from Andrew Doran.

- don't assume the order of cpus in a CPU_INFO_FOREACH loop.
- remove unused structure members.
- simplify.

Current convention is to name/number objects after ci->ci_cpuid, so do
that when creating the kthreads. We may want to change this.

file subr_workqueue.c was added on branch matt-mips64 on 2007-08-05 13:47:26 +0000

Current convention is to name/number objects after ci->ci_cpuid, so do
that when creating the kthreads. We may want to change this.

Improve per-CPU support for the workqueue(9):
 - Make structures CPU-cache friendly, as suggested and explained
   by Andrew Doran.  CACHE_LINE_SIZE definition is invented.
 - Use current CPU if NULL is passed to the workqueue_enqueue().
 - Implemented MI CPU index, which could be used as an index of array.
   Removed linked-lists usage for work queues.

The roundup2() function avoids division, but works only with power of 2.

Reviewed by: <ad>, <yamt>, <tech-kern>

hide internals of struct work.

workqueue_destroy: fix a use-after-free.

Sync with head.

Make MP parts friendly with various ports (especially UP):
 - #ifdef code parts, which uses CPU_INFO_FOREACH/CPU_INFO_ITERATOR
 - use ci_cpuid only in MP case
 - include machine/cpu.h

Implementation of per-CPU work-queues support for workqueue(9) interface.
WQ_PERCPU flag for workqueue and additional argument for workqueue_enqueue()
to assign a CPU might be used. Notes:
 - For now, the list is used for workqueue_queue, which is non-optimal,
   and will be changed with array, where index would be CPU ID.
 - The data structures should be changed to be cache-friendly.

Reviewed by: <yamt>, <tech-kern>

Sync with head.

Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

s/MUTEX_SPIN/MUTEX_DRIVER/ as it's called with IPL_NONE now.

- Pass the error number and residual count to biodone(), and let it handle
  setting error indicators. Prepare to eliminate B_ERROR.
- Add a flag argument to brelse() to be set into the buf's flags, instead
  of doing it directly. Typically used to set B_INVAL.
- Add a "struct cpu_info *" argument to kthread_create(), to be used to
  create bound threads. Change "bool mpsafe" to "int flags".
- Allow exit of LWPs in the IDL state when (l != curlwp).
- More locking fixes & conversion to the new API.

Allow workqueues to be marked as MP safe.

- Add two new arguments to kthread_create1: pri_t pri, bool mpsafe.
- Fork kthreads off proc0 as new LWPs, not new processes.

Eliminate a needless test.

- sync with head.
- move sched_changepri back to kern_synch.c as it doesn't know PPQ anymore.

typedef pri_t and use it instead of int and u_char.

sync with head.

workqueue_exit: update a comment.

use cv_signal rather than cv_broadcast where appropriate.

Merge newlock2 to head.

- Require that cv_signal/cv_broadcast be called with the interlock held.
- Provide 'async' versions that's don't need the interlock.

Eliminate some uses of mtsleep().

Sync with head.

sync with head.

merge yamt-splraiseipl branch.

	- finish implementing splraiseipl (and makeiplcookie).
	  http://mail-index.NetBSD.org/tech-kern/2006/07/01/0000.html
	- complete workqueue(9) and fix its ipl problem, which is reported
	  to cause audio skipping.
	- fix netbt (at least compilation problems) for some ports.
	- fix PR/33218.

sync with head.

Sync with head.

remove some __unused from function parameters.

sync with head

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

don't bother to wrap splraiseipl and IPL_xxx with #if notyet.

adapt to new api.  (#if 0'ed code)

add workqueue_destroy().

workqueue_create: use kmem_alloc rather than malloc.

Use mutexes to protect workqueues.

sync with head

sync with head.

Sync with head.

Merge 2006-05-24 NetBSD-current into the "peter-altq" branch.

sync with head.

sync with head

file subr_workqueue.c was added on branch yamt-lazymbuf on 2006-06-21 15:09:38 +0000

Use for in a forever loop as per KNF.

merge ktrace-lwp.

Sync with HEAD. Here we go again...

sync with head.

file subr_workqueue.c was added on branch ktrace-lwp on 2005-11-10 14:09:45 +0000

file subr_workqueue.c was added on branch yamt-vop on 2005-11-02 11:58:11 +0000

add a simple "do it in thread context" framework.