CVS log for src/sys/kern/kern_sleepq.c

Up to [cvs.NetBSD.org] / src / sys / kern

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

Back out the following revisions on behalf of core:

	sys/sys/lwp.h: revision 1.228
	sys/sys/pipe.h: revision 1.40
	sys/kern/uipc_socket.c: revision 1.306
	sys/kern/kern_sleepq.c: revision 1.84
	sys/rump/librump/rumpkern/locks_up.c: revision 1.13
	sys/kern/sys_pipe.c: revision 1.165
	usr.bin/fstat/fstat.c: revision 1.119
	sys/rump/librump/rumpkern/locks.c: revision 1.87
	sys/ddb/db_xxx.c: revision 1.78
	sys/ddb/db_command.c: revision 1.187
	sys/sys/condvar.h: revision 1.18
	sys/ddb/db_interface.h: revision 1.42
	sys/sys/socketvar.h: revision 1.166
	sys/kern/uipc_syscalls.c: revision 1.209
	sys/kern/kern_condvar.c: revision 1.60

  Add cv_fdrestart() [...]
  Use cv_fdrestart() to implement fo_restart.
  Simplify/streamline pipes a little bit [...]

This changes have caused regressions and need to be debugged.
The cv_fdrestart() addition needs more discussion.

kern_sleepq.c: Sort includes.  No functional change intended.

sys/lwp.h: Nix sys/syncobj.h dependency.

Remove it in ddb/db_syncobj.h too.

New sys/wchan.h defines wchan_t so that users need not pull in
sys/syncobj.h to get it.

Sprinkle #include <sys/syncobj.h> in .c files where it is now needed.

Add cv_fdrestart() (better name suggestions welcome):

Like cv_broadcast(), but make any LWPs that share the same file descriptor
table as the caller return ERESTART when resuming.  Used to dislodge LWPs
waiting for I/O that prevent a file descriptor from being closed, without
upsetting access to the file (not descriptor) made from another direction.

Oops, fix inverted test.

Ensure that an LWP that has taken a legitimate wakeup never produces an
error code from sleepq_block().  Then, it's possible to make cv_signal()
work as expected and only ever wake a singular LWP.

sleepq_block(): slightly reduce number of test+branch in the common case.

sleepq_uncatch(): fix typo that's been there since 2020, hello @thorpej lol:

-       l->l_flag = ~(LW_SINTR | LW_CATCHINTR | LW_STIMO);
+       l->l_flag &= ~(LW_SINTR | LW_CATCHINTR | LW_STIMO);

sleepq_uncatch(): clear LW_STIMO too, so that there's no possibility that
the newly non-interruptable sleep could produce EWOULDBLOCK (paranoia).

Resolve !MULTIPROCESSOR build problem with the nasty kernel lock macros.

Eliminate l->l_biglocks.  Originally I think it had a use but these days a
local variable will do.

Sigh..  Adjust previous to work as intended.  The boosted LWP priority
didn't persist as far as the run queue because l_syncobj gets reset
earlier than I recalled.

- Simplify how priority boost for blocking in kernel is handled.  Rather
  than setting it up at each site where we block, make it a property of
  syncobj_t.  Then, do not hang onto the priority boost until userret(),
  drop it as soon as the LWP is out of the run queue and onto a CPU.
  Holding onto it longer is of questionable benefit.

- This allows two members of lwp_t to be deleted, and mi_userret() to be
  simplified a lot (next step: trim it down to a single conditional).

- While here, constify syncobj_t and de-inline a bunch of small functions
  like lwp_lock() which turn out not to be small after all (I don't know
  why, but atomic_*_relaxed() seem to provoke a compiler shitfit above and
  beyond what volatile does).

kern: KASSERT(A && B) -> KASSERT(A); KASSERT(B)

sleepq(9): Pass syncobj through to sleepq_block.

Previously the usage pattern was:

sleepq_enter(sq, l, lock);              // locks l
...
sleepq_enqueue(sq, ..., sobj, ...);     // assumes l locked, sets l_syncobj
... (*)
sleepq_block(...);			// unlocks l

As long as l remains locked from sleepq_enter to sleepq_block,
l_syncobj is stable, and sleepq_block uses it via ktrcsw to determine
whether the sleep is on a mutex in order to avoid creating ktrace
context-switch records (which involves allocation which is forbidden
in softint context, while taking and even sleeping for a mutex is
allowed).

However, in turnstile_block, the logic at (*) also involves
turnstile_lendpri, which sometimes unlocks and relocks l.  At that
point, another thread can swoop in and sleepq_remove l, which sets
l_syncobj to sched_syncobj.  If that happens, ktrcsw does what is
forbidden -- tries to allocate a ktrace record for the context
switch.

As an optimization, sleepq_block or turnstile_block could stop early
if it detects that l_syncobj doesn't match -- we've already been
requested to wake up at this point so there's no need to mi_switch.
(And then it would be unnecessary to pass the syncobj through
sleepq_block, because l_syncobj would remain stable.)  But I'll leave
that to another change.

Reported-by: syzbot+8b9d7b066c32dbcdc63b@syzkaller.appspotmail.com

ktrace(9): Fix mutex detection in ktrcsw.

On _entry_ to sleepq_block, l->l_syncobj is set so that ktrcsw
(ktr_csw) has the opportunity to detect whether it's a mutex or
rwlock.  It is critical to avoid ktealloc when we're sleeping on a
mutex because we may be in softint context where ktealloc is
forbidden.

But after mi_switch, on _exit_ from sleepq_block, l->l_syncobj may
have been changed back to &sched_syncobj or something by
sleepq_remove, and so ktrcsw can no longer rely on l->l_syncobj to
determine whether we _were_ sleeping on a mutex or not.

Instead, save the syncobj in sleepq_block and pass it through as an
argument to ktrcsw.

Reported-by: syzbot+414edba9d161b7502658@syzkaller.appspotmail.com
Reported-by: syzbot+4425c97ac717b12495a2@syzkaller.appspotmail.com
Reported-by: syzbot+5812565b926ee8eb5cf3@syzkaller.appspotmail.com
Reported-by: syzbot+8b9d7b066c32dbcdc63b@syzkaller.appspotmail.com
Reported-by: syzbot+909a8e743c967d97f433@syzkaller.appspotmail.com
Reported-by: syzbot+e2a34bb5509bea0bba11@syzkaller.appspotmail.com
Reported-by: syzbot+faaea3aad6c9d0829f76@syzkaller.appspotmail.com

fix various typos, mainly in comments, but also log messages, docs, game text.

s/happends/happens/ in comment.

- sleepq_block(): Add a new LWP flag, LW_CATCHINTR, that is used to track
  the intent to catch signals while sleeping.  Initialize this flag based
  on the catch_p argument to sleepq_block(), and rather than test catch_p
  when awakened, test LW_CATCHINTR.  This allows the intent to change
  (based on whatever criteria the owner of the sleepq wishes) while the
  LWP is asleep.  This is separate from LW_SINTR in order to leave all
  other logic around LW_SINTR unaffected.
- In sleepq_transfer(), adjust also LW_CATCHINTR based on the catch_p
  argument.  Also allow the new LWP lock argument to be NULL, which
  will cause the lwp_setlock() call to be skipped; this allows transfer
  to another sleepq that is known to be protected by the same lock.
- Add a new function, sleepq_uncatch(), that will transition an LWP
  from "interruptible sleep" to "uninterruptible sleep" on its current
  sleepq.

In sleepq_insert(), in the SOBJ_SLEEPQ_SORTED case, if there are existing
waiters of lower priority, then the new LWP will be inserted in FIFO order
with respect to other LWPs of the same priority.  However, if all other
LWPs are of equal priority to the LWP being inserted, the new LWP would
be inserted in LIFO order.

Fix this to always insert in FIFO order with respect to equal priority LWPs.

OK ad@.

Add a new function, sleepq_transfer(), that moves an lwp from one
sleepq to another.

Sync with HEAD

Sync with HEAD

Set LW_SINTR earlier so it doesn't pose a problem for doing interruptable
waits with turnstiles (not currently done).

hardclock_ticks -> getticks()

Mostly merge changes from HEAD upto 20200411

- Make this needed sequence always work for condvars, by not touching the CV
  again after wakeup.  Previously it could panic because cv_signal() could
  be called by cv_wait_sig() + others:

	cv_broadcast(cv);
	cv_destroy(cv);

- In support of the above, if an LWP doing a timed wait is awoken by
  cv_broadcast() or cv_signal(), don't return an error if the timer
  fires after the fact, i.e. either succeed or fail, not both.

- Remove LOCKDEBUG code for CVs which never worked properly and is of
  questionable use.

Merge changes from current as of 20200406

Change sleepq_t from a TAILQ to a LIST and remove SOBJ_SLEEPQ_FIFO.  Only
select/poll used the FIFO method and that was for collisions which rarely
occur.  Shrinks sleep_t and condvar_t.

Update a comment.

Sync with head.

- Move the LW_RUNNING flag back into l_pflag: updating l_flag without lock
  in softint_dispatch() is risky.  May help with the "softint screwup"
  panic.

- Correct the memory barriers around zombies switching into oblivion.

fix incorrect type

Add SOBJ_SLEEPQ_NULL: means there is no TAILQ and the caller tracks the
sleeping LWPs some other way, which sleepq_*() doesn't know about.

Sync with head.

Nothing uses l->l_sleeperr any more.

Hopefully fix some problems seen with MP support on non-x86, in particular
where curcpu() is defined as curlwp->l_cpu:

- mi_switch(): undo the ~2007ish optimisation to unlock curlwp before
  calling cpu_switchto().  It's not safe to let other actors mess with the
  LWP (in particular l->l_cpu) while it's still context switching.  This
  removes l->l_ctxswtch.

- Move the LP_RUNNING flag into l->l_flag and rename to LW_RUNNING since
  it's now covered by the LWP's lock.

- Ditch lwp_exit_switchaway() and just call mi_switch() instead.  Everything
  is in cache anyway so it wasn't buying much by trying to avoid saving old
  state.  This means cpu_switchto() will never be called with prevlwp ==
  NULL.

- Remove some KERNEL_LOCK handling which hasn't been needed for years.

Fix LOCKDEBUG panic on mutex_init().

Reported-by: syzbot+5a77339dc0a55e8d8caa@syzkaller.appspotmail.com

As with turnstiles, don't bother allocating sleepq locks with mutex_obj_alloc(),
and avoid the indirect reference.

Make it possible to call mi_switch() and immediately switch to another CPU.
This seems to take about 3us on my Intel system.  Two changes required:

- Have the caller to mi_switch() be responsible for calling spc_lock().
- Avoid using l->l_cpu in mi_switch().

While here:

- Add a couple of calls to membar_enter()
- Have the idle LWP set itself to LSIDL, to match softint_thread().
- Remove unused return value from mi_switch().

Minor scheduler cleanup:

- Adapt to cpu_need_resched() changes. Avoid lost & duplicate IPIs and ASTs.
  sched_resched_cpu() and sched_resched_lwp() contain the logic for this.
- Changes for LSIDL to make the locking scheme match the intended design.
- Reduce lock contention and false sharing further.
- Numerous small bugfixes, including some corrections for SCHED_FIFO/RT.
- Use setrunnable() in more places, and merge cut & pasted code.

Sleep queues & turnstiles:

- Avoid false sharing.
- Make the turnstile hash function more suitable.
- Increase turnstile hash table size.
- Make amends by having only one set of system wide sleep queue hash locks.

update from HEAD

Sync with HEAD

GSoC 2016 Charles Cui: Implement thread priority protection based on work
by Andy Doran. Also document the get/set pshared thread calls as not
implemented, and add a skeleton implementation that is disabled.
XXX: document _sched_protect(2).

Don't next structure and enum definitions.
Don't use C++ keywords new, try, class, private, etc.

Rebase to HEAD as of a few days ago.

Rebase.

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs.  ("Protocol error: too many arguments")

sync with head

Make sleepq_wake() type void.  The return value hasn't been used in
almost 6 years.  Even if it were, returning an arbitrary lwp is a bit
of a wonky interface and can really work only when expected == 1.

resync from head

Add comments saying that a cv_timedwait and sleepq_block interpret
timo = 0 as an infinite timeout.  This is already documented in the
cv_timedwait(9) man page, and there is no sleeq_block(9) man page.

sync with head

Remove safepri and use IPL_SAFEPRI instead.  This may be defined in a MD
header file (if not, a value of 0 is assmued).

sync with head

sync to -current.

Remove COMPAT_SA / KERN_SA.  Welcome to 6.99.3!
Approved by core@.

merge to -current.

Apply patch, requested by jmcneill in ticket #1668:
	sys/arch/amd64/amd64/syscall.c			patch
	sys/arch/i386/i386/syscall.c			patch
	sys/arch/i386/i386/trap.c			patch
	sys/kern/kern_sig.c				patch
	sys/kern/kern_sleepq.c				patch
	sys/kern/kern_subr.c				patch
	sys/kern/sys_process.c				patch
	sys/secmodel/bsd44/secmodel_bsd44_suser.c	patch
	sys/sys/proc.h					patch
	sys/sys/ptrace.h				patch

arch/i386/i386/machdep.c, arch/amd64/amd64/machdep.c (from
arch/x86/x86/machdep.c) by christos:
Remove code that was used to avoid register spills. setcontext(2) can change
the registers, so re-fetching will produce the wrong result for trace_exit().
arch/i386/i386/trap.c by reinoud:
Fix the illegal instruction return address. It was using the value of the
cpu's %cr2 register but thats not valid:

CR2 Contains a value called Page Fault Linear Address (PFLA). When a page
fault occurs, the address the program attempted to access is stored in the CR2
register.

And this is thus NOT the illegal instruction address!
kern/kern_sig.c by christos:
PR kern/45327: Jared McNeill: ptrace: siginfo doesn't work with traced processes
When saving the signal in p->p_xstat, clear it from the pending mask, but
don't remove it from the siginfo queue, so that next time the debugger
delivers it, the original information is found.
When posting a signal from the debugger l->l_sigpendset is not set, so we
use the process pending signal and add it back to the process pending set.
Split sigget into sigget() and siggetinfo(). When a signal comes from the
debugger (l->l_sigpendset == NULL), using siggetinfo() try to fetch the
siginfo information from l->l_sigpend and then from p->p_sigpend if it
was not found. This allows us to pass siginfo information for traps from
the debugger.
don't delete signal from the debugger.
kern/kern_sleepq.c by christos:
PR kern/40594: Antti Kantee: Don't call issignal() here to determine what errno
to set for the interrupted syscall, because issignal() will consume the signal
and it will not be delivered to the process afterwards. Instead call
sigispending() (which now returns the first pending signal) and does not
consume the signal.
We need to process SA_STOP signals immediately, and not deliver them to
the process. Instead of re-structuring the code to do that, call issignal()
like before in that case. (tail -F /file^Zfg should not get interrupted).
kern/kern_subr.c by jmcneill, christos:
PR kern/45312: ptrace: PT_SETREGS can't alter system calls

Add a new PT_SYSCALLEMU request that cancels the current syscall, for
use with PT_SYSCALL.
For PT_SYSCALLEMU, no need to stop again on syscall exit.
ifdef unused variable with -UPTRACE

kern/sys_process.c, sys/proc.h, sys/ptrace.h, secmodel/bsd44/secmodel_bsd44_suser.c by jmcneill, christos:
PR kern/43681: PT_SYSCALL appears to be broken

sys_ptrace: For PT_CONTINUE/PT_SYSCALL/PT_DETACH, modify the p_trace_enabled
flag of the target process, not the calling process.
Process the signal now, otherwise calling issignal() and ignoring
the return will lose the signal if it came from the debugger
(issignal() clears p->p_xstat)
PR kern/45312: ptrace: PT_SETREGS can't alter system calls

Add a new PT_SYSCALLEMU request that cancels the current syscall, for
use with PT_SYSCALL.
PR kern/45330: ptrace: signals can alter syscall return values

process_stoptrace: defer signal processing to userret, ok christos@

Remove obsolete ltsleep(9) and wakeup_one(9).

- make lendpri/changepri similar.
- make common code a subroutine.

We need to process SA_STOP signals immediately, and not deliver them to
the process. Instead of re-structuring the code to do that, call issignal()
like before in that case. (tail -F /file^Zfg should not get interrupted).

PR/40594: Antti Kantee: Don't call issignal() here to determine what errno
to set for the interrupted syscall, because issignal() will consume the signal
and it will not be delivered to the process afterwards. Instead call
sigispending() (which now returns the first pending signal) and does not
consume the signal.

These don't need uvm/uvm_extern.h.

sleepq_insert: call lwp_eprio only when necessary

Sync with HEAD.

sync with head

Sprinkle __cacheline_aligned and __read_mostly, make some functions static.

drop inline here, to avoid C99 vs GNU differences

sync with head

Remove uarea swap-out functionality:

- Addresses the issue described in PR/38828.
- Some simplification in threading and sleepq subsystems.
- Eliminates pmap_collect() and, as a side note, allows pmap optimisations.
- Eliminates XS_CTL_DATA_ONSTACK in scsipi code.
- Avoids few scans on LWP list and thus potentially long holds of proc_lock.
- Cuts ~1.5k lines of code.  Reduces amd64 kernel size by ~4k.
- Removes __SWAP_BROKEN cases.

Tested on x86, mips, acorn32 (thanks <mpumford>) and partly tested on
acorn26 (thanks to <bjh21>).

Discussed on <tech-kern>, reviewed by <ad>.

Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.

sync with head.

Sync with HEAD.

Allocate sleep queue locks with mutex_obj_alloc. Reduces memory usage
on !MP kernels, and reduces false sharing on MP ones.

Sync with HEAD.

Sync with HEAD.

Merge wrstuden-revivesa into HEAD.

Sync with HEAD.

Sync with wrstuden-revivesa-base-2.

sleepq_block: fix a bug to lose biglocks in the case of recursive calls.

this fixes pf rb-tree corruption on my box.

Add support for compiling SA as an option. Implied by COMPAT_40.

i386 kernels both with COMPAT_40 and with no compat options (and thus
no SA) compile.

No functional changes intended.

Change how we make SA threads not generate upcalls. Instead of clearing
LW_SA, use a private flag, LP_SA_NOBLOCK, that we set when we want
to not generate upcalls. This means we do NOT need to lock (l)
(ourselves) to set it.

Adjust tests that look at LW_SA. Now, we are an upcall-generating
lwp if ((l->l_flag & LW_SA) && (~l->l_pflag & LP_SA_NOBLOCK)).

Introduce code pattern to set & remember this:

	f = ~l->l_pflag & LP_SA_NOBLOCK;
	l->l_pflag |= LP_SA_NOBLOCK;

	...

	/* f is now LP_SA_NOBLOCK if it wasn't set in l_pflag before */

	l->l_pflag ^= f;

I updated a lot of the trap handlers to do trap handling iff LP_SA_NOBLOCK
is not set. I tried to figure out if the trap handler could be triggered
for user-based faults as opposed to kernel faults to user addresses, and
only look at LP_SA_NOBLOCK for the latter.

Above is a result of discussions with rmind at to reduce lock twiddling.

Also, per same discussions, add locking to sys_sa_preempt(). p_lock is
the lock we want.

Also, per same discussions, remove use of LSSUSPENDED as a thread state.
We needed to use it when we were emulating the 4.X and previous behavior
of hiding cached threads. For the moment, we now have them instead
remain visible to all and have them sleeping on the "lwpcache" wait
channel.

sa_newcachelwp(): sa_putcachelwp() wants savp_mutex held, not p_lock.

Tweak some comments.

Sync with HEAD.

Sync w/ -current. 34 merge conflicts to follow.

Sync with head.

sync with head.

sleepq_block: add a comment.

PR kern/38761: new (?) race in buffer cache code

sleepq_changepri, sleepq_lendpri: don't let an active sleep queue head become
empty. The condvar code inspects the queue head without holding the sleep
queue lock and needs to see a non-empty queue if there are waiters.

sync with head

Sync with HEAD.

PR kern/38812 race between lwp_exit_switchaway and exit1/coredump

Move the LWP RUNNING and TIMEINTR flags into the thread-private flag word.

Take the mutex pointer and waiters count out of sleepq_t: the values can
be or are maintained elsewhere. Now a sleepq_t is just a TAILQ_HEAD.

Re-add calls to sa_awaken() and sa_switch(). Also add
back sa_awaken() itself. While here, remove the "type"
arguement to sa_switch() - it is all about generating
blocked upcalls.

- Make periodical balancing mandatory.
- Fix priority raising in M2 (broken after making runqueues mandatory).

sync with head.

sync with head.

Remove clause 3 and 4 from TNF licenses

Merge proc::p_mutex and proc::p_smutex into a single adaptive mutex, since
we no longer need to guard against access from hardware interrupt handlers.

Additionally, if cloning a process with CLONE_SIGHAND, arrange to have the
child process share the parent's lock so that signal state may be kept in
sync. Partially addresses PR kern/37437.

Give callout_halt() an additional 'kmutex_t *interlock' argument. If there
is a need to block and wait for the callout to complete, and there is an
interlock, it will be dropped while waiting and reacquired before return.

Take the run queue management code from the M2 scheduler, and make it
mandatory. Remove the 4BSD run queue code. Effects:

- Pluggable scheduler is only responsible for co-ordinating timeshared jobs.
- All systems run with per-CPU run queues.
- 4BSD scheduler gets processor sets / affinity.
- 4BSD scheduler gets a significant peformance boost on some workloads.

Discussed on tech-kern@.

assertions.

Sync with HEAD.

sleepq_block: use callout_halt, as we have to wait for the callout to
stop (it might be running on another CPU). Otherwise, 'curlwp' could
exit before it completes.

sync with head.

sync with head.

sync with HEAD

Add a boolean parameter to syncobj_t::sobj_unsleep. If true we want the
existing behaviour: the unsleep method unlocks and wakes the swapper if
needs be. If false, the caller is doing a batch operation and will take
care of that later. This is kind of ugly, but it's difficult for the caller
to know which lock to release in some situations.

sync with head.

Sync with HEAD.

Make schedstate_percpu::spc_lwplock an exernally allocated item. Remove
the hacks in sparc/cpu.c to reinitialize it. This should be in its own
cache line but that's another change.

sync with head

sync with HEAD

Sync with HEAD

Start detangling lock.h from intr.h. This is likely to cause short term
breakage, but the mess of dependencies has been regularly breaking the
build recently anyhow.

Sync with HEAD.

Sync with HEAD.

Sync with head.

sync with head

Match the docs: MUTEX_DRIVER/SPIN are now only for porting code written
for Solaris.

Sync with HEAD.

sync with head.

Sync with HEAD

sync with HEAD

Sync with HEAD.

Merge scheduler changes from the vmlocking branch. All discussed on
tech-kern:

- Invert priority space so that zero is the lowest priority. Rearrange
  number and type of priority levels into bands. Add new bands like
  'kernel real time'.
- Ignore the priority level passed to tsleep. Compute priority for
  sleep dynamically.
- For SCHED_4BSD, make priority adjustment per-LWP, not per-process.

- Fix interactivity problems under high load. Beacuse soft interrupts
  are being stacked on top of regular LWPs, more often than not aston()
  was being called on a soft interrupt thread instead of a user thread,
  meaning that preemption was not happening on EOI.

- Don't use bool in a couple of data structures. Sub-word writes are not
  always atomic and may clobber other fields in the containing word.

- For SCHED_4BSD, make p_estcpu per thread (l_estcpu). Rework how the
  dynamic priority level is calculated - it's much better behaved now.

- Kill the l_usrpri/l_priority split now that priorities are no longer
  directly assigned by tsleep(). There are three fields describing LWP
  priority:

        l_priority: Dynamic priority calculated by the scheduler.
                This does not change for kernel/realtime threads,
                and always stays within the correct band. Eg for
                timeshared LWPs it never moves out of the user
                priority range. This is basically what l_usrpri
                was before.

        l_inheritedprio: Lent to the LWP due to priority inheritance
                (turnstiles).

        l_kpriority: A boolean value set true the first time an LWP
                sleeps within the kernel. This indicates that the LWP
                should get a priority boost as compensation for blocking.
                lwp_eprio() now does the equivalent of sched_kpri() if
                the flag is set. The flag is cleared in userret().

- Keep track of scheduling class (OTHER, FIFO, RR) in struct lwp, and use
  this to make decisions in a few places where we previously tested for a
  kernel thread.

- Partially fix itimers and usr/sys/intr time accounting in the presence
  of software interrupts.

- Use kthread_create() to create idle LWPs. Move priority definitions
  from the various modules into sys/param.h.

- newlwp -> lwp_create

sync with head.

Sync with HEAD.

Follow the merge of pmap.c on i386 and amd64 and move
pmap_init_tmp_pgtbl into arch/x86/x86/pmap.c. Modify the ACPI wakeup
code to restore CR4 before jumping back into kernel space as the large
page option might cover that.

Sync with head.

sync with head.

Restore previous wakeup/preemption behaviour. May fix problems with gxine.
This has very little effect with SCHED_M2, possibly because threads are less
likely to migrate, and because a number of issues with contention on user
space objects has been fixed.

Fix scheduler priority lossage. From rmind@.

sleepq_remove: remove a stale comment.

sync with head.

Sync with HEAD (pull-up a fix).

sleepq_remove: Do not call sched_wakeup() when thread is running.
This fixes a locking problem, when l_cpu is changed in LSONPROC state.
Possible case was noted by <ad>.

Sync with HEAD.

Import of SCHED_M2 - the implementation of new scheduler, which is based
on the original approach of SVR4 with some inspirations about balancing
and migration from Solaris.  It implements per-CPU runqueues, provides a
real-time (RT) and time-sharing (TS) queues, ready to support a POSIX
real-time extensions, and also prepared for the support of CPU affinity.

The following lines in the kernel config enables the SCHED_M2:

no options SCHED_4BSD
options SCHED_M2

The scheduler seems to be stable.  Further work will come soon.

http://mail-index.netbsd.org/tech-kern/2007/10/04/0001.html
http://www.netbsd.org/~rmind/m2/mysql_bench_ro_4x_local.png
Thanks <ad> for the benchmarks!

Sync with head.

Sync with head.

Sync with HEAD.

Sync with HEAD.

Sync with _lwp_park/priority changes on HEAD.

- Fix sleepq_block() to return EINTR if the LWP is cancelled. Pointed out
  by yamt@.

- Introduce SOBJ_SLEEPQ_LIFO, and use for LWPs sleeping via _lwp_park.
  libpthread enqueues most waiters in LIFO order to try and wake LWPs that
  ran recently, since their working set is more likely to be in cache.
  Matching the order of insertion reduces the time spent searching queues
  in the kernel.

- Do not boost the priority of LWPs sleeping in _lwp_park, just let them
  sleep at their user priority level. LWPs waiting for some I/O event in
  the kernel still wait with kernel priority and get woken more quickly.
  This needs more evaluation and is to be revisited, but the effect on a
  variety of benchmarks is positive.

- When waking LWPs, do not send an IPI to remote CPUs or arrange for the
  current LWP to be preempted unless (a) the thread being awoken has kernel
  priority and has higher priority than the currently running thread or (b)
  the remote CPU is idle.

Sync with HEAD.

sync with head.

Sync with HEAD.

pull the following change from vmlocking branch.

	revision 1.7.2.10
	date: 2007/08/27 12:51:13;  author: yamt;  state: Exp;  lines: +6 -7
	sleepq_block: don't call lwp_unsleep twice.
	(fix an assertion failure in lwp_unsleep.)

When awakening a thread on a remote CPU, if it has kernel priority or higher
then use RESCHED_IMMED. This is important for rwlocks, since they do direct
handoff. If the owner does not run soon then other threads trying to acquire
the lock will back up waiting for the running thread to be preempted due to
some other event, and the lock can become a chokepoint. I suspect this may
also fix a loss of interactivity when the boot CPU (or whichever CPU is
handling most device interrupts) is taken offline with cpuctl on a busy
system.

sleepq_block: don't call lwp_unsleep twice.
(fix an assertion failure in lwp_unsleep.)

sleepq_block: fix a merge botch.  (s/error/early/)

Sync with HEAD.

Sync with HEAD.

Sync with HEAD.

Changes to make ktrace LKM friendly and reduce ifdef KTRACE. Proposed
on tech-kern.

sleepq_block: if a pending signal is detected but has already been taken
by the time the calling thread tries to take it, don't return EINTR.
Instead return zero leading to a spurious wakeup.

file kern_sleepq.c was added on branch matt-mips64 on 2007-08-01 23:30:55 +0000

sleepq_block: if a pending signal is detected but has already been taken
by the time the calling thread tries to take it, don't return EINTR.
Instead return zero leading to a spurious wakeup.

Sync with head.

Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

Adapt to callout API change.

- Increase the number of thread priorities from 128 to 256. How the space
  is set up is to be revisited.
- Implement soft interrupts as kernel threads. A generic implementation
  is provided, with hooks for fast-path MD code that can run the interrupt
  threads over the top of other threads executing in the kernel.
- Split vnode::v_flag into three fields, depending on how the flag is
  locked (by the interlock, by the vnode lock, by the file system).
- Miscellaneous locking fixes and improvements.

Sync with head.

merge yamt-idlelwp branch.  asked by core@.  some ports still needs work.

from doc/BRANCHES:

	idle lwp, and some changes depending on it.

	1. separate context switching and thread scheduling.
	   (cf. gmcgarry_ctxsw)
	2. implement idle lwp.
	3. clean up related MD/MI interfaces.
	4. make scheduler(s) modular.

Assign a per-CPU lock to LWPs as they transition into the ONPROC state.

http://mail-index.netbsd.org/tech-kern/2007/05/06/0003.html

Some changes mainly for top/ps:

- Add an optional name field to struct lwp.
- Count the total number of context switches + involuntary,
  not voluntary + involuntary.
- Mark the idle threads as LSIDL when not running, otherwise
  they show up funny in a top(1) that shows threads.
- Make pctcpu and cpticks per-LWP attributes.
- Add to kinfo_lwp: cpticks, pctcpu, pid, name.

Pull up a change from the vmlocking branch:

- Ensure that LWPs going to sleep are on the sleep queue before releasing
  any interlocks. This is so that calls to turnstile_wakeup will have the
  correct locks held when adjusting priority. Avoids another deadlock.
- Assume that LWPs blocked on a turnstile will never be swapped out.
- LWPs blocking on a turnstile must have kernel priority, as they
  are consuming kernel resources.

- Nuke the seperate scheduler locking scheme for UP kernels - it has been
  at the root of too many bugs.
- Add a LW_BOUND flag that indicates an LWP is bound to a specific CPU.

sync with head.

- Ensure that that LWPs going to sleep are on the sleep queue and so
  have their syncobj pointer updated, so that calls to turnstile_wakeup
  will have the correct locks held when adjusting the current LWP's
  priority. Avoids another deadlock.
- Assume that LWPs blocked on a turnstile will never be swapped out.
- LWPs blocking on a turnstile must have kernel priority, as they
  are consuming kernel resources.

Sync with head.

kernel_lock isn't needed for memory allocation any more.

- Make context switch counters 64-bit, and count the total number of
  context switches + voluntary, instead of involuntary + voluntary.
- Add lwp::l_swaplock for uvm.
- PHOLD/PRELE are replaced.

Pullup to -current

- cv_wakeup: remove this. There are ~zero situations where it's useful.
- cv_wait and friends: after resuming execution, check to see if we have
  been restarted as a result of cv_signal. If we have, but cannot take
  the wakeup (because of eg a pending Unix signal or timeout) then try to
  ensure that another LWP sees it. This is necessary because there may
  be multiple waiters, and at least one should take the wakeup if possible.
  Prompted by a discussion with pooka@.
- typedef struct lwp lwp_t;
- int -> bool, struct lwp -> lwp_t in a few places.

Fix UP kernels. Having the different locking scheme for UP/MP is turning
out to be more hassle than it's worth...

Checkpoint:
- Abstract for per-CPU locking of runqueues.
  As a workaround for SCHED_4BSD global runqueue, covered by sched_mutex,
  spc_mutex is a pointer for now. After making SCHED_4BSD runqueues
  per-CPU, it will became a storage mutex.
- suspendsched: Locking is not necessary for cpu_need_resched().
- Remove mutex_spin_exit() prototype in patch.c and LOCK_ASSERT() check
  in runqueue_nextlwp() in sched_4bsd.c to make them compile again.

remove a stale updatepri prototype.

Previously cpu_info::ci_curlwp was protected by the sched_mutex and this is
used in a few places to synchronise. Now the state of LWPs is protected
during switch by their current lock (which might be e.g. a sleep queue
lock). So ci_curlwp is unlocked, which is necessary to be able to do
preemption and to run interrupts as LWPs cheaply.

Add a (locked) flag to the lwp (LW_RUNNING) that indicates if it is on CPU
somewhere. More exactly, it means that the LWP's state is tied to a CPU, and
that the LWP has not yet switched away even if (l->l_cpu->ci_curcpu != l) or
(l->l_stat != LSONPROC).

Do not do an implicit enqueue in sched_switch(), move enqueueing back to
the dispatcher. Rename sched_switch() back to sched_nextlwp(). Add for
sched_enqueue() new argument, which indicates the calling from mi_switch().

Requested by yamt@

Checkpoint:

- Addition of scheduler-specific pointers in the struct proc, lwp and
  schedstate_percpu.
- Addition of sched_lwp_fork(), sched_lwp_exit() and sched_slept() hooks.
- mi_switch() now has only one argument.
- sched_nextlwp(void) becomes sched_switch(struct lwp *) and does an
  enqueueing of LWP.
- Addition of general kern.sched sysctl node.
- Remove twice called uvmexp.swtch++, other cleanups.

Discussed on tech-kern@

- sync with head.
- move sched_changepri back to kern_synch.c as it doesn't know PPQ anymore.

typedef pri_t and use it instead of int and u_char.

implement priority inheritance.

add an "immediate" flag for cpu_need_resched().  suggested by Andrew Doran.

sync with head.

- introduce sys/cpu.h which has cpu_idle and cpu_need_resched.
- use it where appropriate.
- while i'm here, remove several unnecessary #include.

General Common Scheduler Framework (CSF) patch import. Huge thanks for
Daniel Sieger <dsieger at TechFak.Uni-Bielefeld de> for this work.

Short abstract: Split the dispatcher from the scheduler in order to
make the scheduler more modular. Introduce initial API for other
schedulers' implementations.

Discussed in tech-kern@
OK: yamt@, ad@

Note: further work will go soon.

file kern_sleepq.c was added on branch yamt-lazymbuf on 2007-02-26 09:11:11 +0000

Change the process/lwp flags seen by userland via sysctl back to the
P_*/L_* naming convention, and rename the in-kernel flags to avoid
conflict. (P_ -> PK_, L_ -> LW_ ). Add back the (now unused) LSDEAD
constant.

Restores source compatibility with pre-newlock2 tools like ps or top.

Reviewed by Andrew Doran.

Add uvm_kick_scheduler() (MP safe) to replace wakeup(&proc0).

remove function prototypes of sa_awaken.

Merge newlock2 to head.

- Change syncobj_t::sobj_changepri() to alter both the user priority and
  the effective priority of LWPs. How the effective priority is adjusted
  depends on the type of object.
- Add a couple of comments to sched_kpri() and remrunqueue().

Try to reduce cache line ping-ponging.

sleepq_changepri(): don't forget to change l_priority..

Push the kernel_lock back in a couple of places. Sleep/wakeup and ktrace
are now MP safe.

Remove support for SA. Ok core@.

Rename some functions to better describe what they do.

sleepq_timeout(): need to use lwp_unsleep() here.

Minor corrections.

Checkpoint work in progress.

Fix an obvious sleep/wakeup bug introduced in previous.

Checkpoint work in progress.

- Redo LWP locking slightly and fix some races.
- Fix some locking botches.
- Make signal mask / stack per-proc for SA processes.
- Add _lwp_kill().

sleepq_sigtoerror: fix a locking botch.

sleepq_block(): acquire sched_mutex before calling sleepq_remove()

Add a sleep queue implementation.

file kern_sleepq.c was initially added on branch newlock2.