CVS log for src/sys/arch/arm/arm32/cpuswitch.S

Up to [cvs.NetBSD.org] / src / sys / arch / arm / arm32

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

Pull up following revision(s) (requested by riastradh in ticket #1859):

	sys/arch/ia64/ia64/vm_machdep.c: revision 1.18
	sys/arch/powerpc/powerpc/locore_subr.S: revision 1.67
	sys/arch/aarch64/aarch64/locore.S: revision 1.91
	sys/arch/mips/include/asm.h: revision 1.74
	sys/arch/hppa/include/cpu.h: revision 1.13
	sys/arch/arm/arm/armv6_start.S: revision 1.38
	 (applied also to sys/arch/arm/cortex/a9_mpsubr.S,
	 sys/arch/arm/cortex/a9_mpsubr.S,
	 sys/arch/arm/cortex/cortex_init.S)
	sys/arch/evbmips/ingenic/cpu_startup.S: revision 1.2
	sys/arch/mips/mips/locore.S: revision 1.229
	sys/arch/alpha/include/asm.h: revision 1.45
	 (applied to sys/arch/alpha/alpha/multiproc.s)
	sys/arch/sparc64/sparc64/locore.s: revision 1.432
	sys/arch/vax/vax/subr.S: revision 1.42
	sys/arch/mips/mips/locore_mips3.S: revision 1.116
	sys/arch/ia64/ia64/machdep.c: revision 1.44
	sys/arch/arm/arm32/cpuswitch.S: revision 1.106
	sys/arch/sparc/sparc/locore.s: revision 1.284
	(all via patch)

aarch64: Add missing barriers in cpu_switchto.
Details in comments.

Note: This is a conservative change that inserts a barrier where
there was a comment saying none is needed, which is probably correct.
The goal of this change is to systematically add barriers to be
confident in correctness; subsequent changes may remove some bariers,
as an optimization, with an explanation of why each barrier is not
needed.

PR kern/57240

alpha: Add missing barriers in cpu_switchto.
Details in comments.

arm32: Add missing barriers in cpu_switchto.
Details in comments.

hppa: Add missing barriers in cpu_switchto.
Not sure hppa has ever had working MULTIPROCESSOR, so maybe no
pullups needed?

ia64: Add missing barriers in cpu_switchto.
(ia64 has never really worked, so no pullups needed, right?)

mips: Add missing barriers in cpu_switchto.
Details in comments.

powerpc: Add missing barriers in cpu_switchto.
Details in comments.

sparc: Add missing barriers in cpu_switchto.

sparc64: Add missing barriers in cpu_switchto.
Details in comments.

vax: Note where cpu_switchto needs barriers.

Not sure vax has ever had working MULTIPROCESSOR, though, and I'm not
even sure how to spell store-before-load barriers on VAX, so no
functional change for now.

Pull up following revision(s) (requested by riastradh in ticket #1676):

	sys/arch/ia64/ia64/vm_machdep.c: revision 1.18
	sys/arch/powerpc/powerpc/locore_subr.S: revision 1.67
	sys/arch/aarch64/aarch64/locore.S: revision 1.91
	sys/arch/mips/include/asm.h: revision 1.74
	sys/arch/hppa/include/cpu.h: revision 1.13
	sys/arch/arm/arm/armv6_start.S: revision 1.38
	sys/arch/evbmips/ingenic/cpu_startup.S: revision 1.2
	sys/arch/mips/mips/locore.S: revision 1.229
	sys/arch/aarch64/aarch64/cpuswitch.S: revision 1.40
	sys/arch/alpha/include/asm.h: revision 1.45
	sys/arch/sparc64/sparc64/locore.s: revision 1.432
	sys/arch/vax/vax/subr.S: revision 1.42
	sys/arch/mips/mips/locore_mips3.S: revision 1.116
	sys/arch/ia64/ia64/machdep.c: revision 1.44
	sys/arch/arm/arm32/cpuswitch.S: revision 1.106
	sys/arch/sparc/sparc/locore.s: revision 1.284
	(all via patch)

aarch64: Add missing barriers in cpu_switchto.
Details in comments.

Note: This is a conservative change that inserts a barrier where
there was a comment saying none is needed, which is probably correct.
The goal of this change is to systematically add barriers to be
confident in correctness; subsequent changes may remove some bariers,
as an optimization, with an explanation of why each barrier is not
needed.

PR kern/57240

alpha: Add missing barriers in cpu_switchto.
Details in comments.

arm32: Add missing barriers in cpu_switchto.
Details in comments.

hppa: Add missing barriers in cpu_switchto.
Not sure hppa has ever had working MULTIPROCESSOR, so maybe no
pullups needed?

ia64: Add missing barriers in cpu_switchto.
(ia64 has never really worked, so no pullups needed, right?)

mips: Add missing barriers in cpu_switchto.
Details in comments.

powerpc: Add missing barriers in cpu_switchto.
Details in comments.

sparc: Add missing barriers in cpu_switchto.

sparc64: Add missing barriers in cpu_switchto.
Details in comments.

vax: Note where cpu_switchto needs barriers.

Not sure vax has ever had working MULTIPROCESSOR, though, and I'm not
even sure how to spell store-before-load barriers on VAX, so no
functional change for now.

Pull up following revision(s) (requested by riastradh in ticket #264):

	sys/arch/ia64/ia64/vm_machdep.c: revision 1.18
	sys/arch/powerpc/powerpc/locore_subr.S: revision 1.67
	sys/arch/aarch64/aarch64/locore.S: revision 1.91
	sys/arch/mips/include/asm.h: revision 1.74
	sys/arch/hppa/include/cpu.h: revision 1.13
	sys/arch/arm/arm/armv6_start.S: revision 1.38
	sys/arch/evbmips/ingenic/cpu_startup.S: revision 1.2
	sys/arch/mips/mips/locore.S: revision 1.229
	sys/arch/aarch64/aarch64/cpuswitch.S: revision 1.40
	sys/arch/alpha/include/asm.h: revision 1.45
	sys/arch/sparc64/sparc64/locore.s: revision 1.432
	sys/arch/vax/vax/subr.S: revision 1.42
	sys/arch/mips/mips/locore_mips3.S: revision 1.116
	sys/arch/riscv/riscv/cpu_switch.S: revision 1.3
	sys/arch/ia64/ia64/machdep.c: revision 1.44
	sys/arch/arm/arm32/cpuswitch.S: revision 1.106
	sys/arch/sparc/sparc/locore.s: revision 1.284

aarch64: Add missing barriers in cpu_switchto.
Details in comments.

Note: This is a conservative change that inserts a barrier where
there was a comment saying none is needed, which is probably correct.
The goal of this change is to systematically add barriers to be
confident in correctness; subsequent changes may remove some bariers,
as an optimization, with an explanation of why each barrier is not
needed.

PR kern/57240

alpha: Add missing barriers in cpu_switchto.
Details in comments.

arm32: Add missing barriers in cpu_switchto.
Details in comments.

hppa: Add missing barriers in cpu_switchto.
Not sure hppa has ever had working MULTIPROCESSOR, so maybe no
pullups needed?

ia64: Add missing barriers in cpu_switchto.
(ia64 has never really worked, so no pullups needed, right?)

mips: Add missing barriers in cpu_switchto.
Details in comments.

powerpc: Add missing barriers in cpu_switchto.
Details in comments.

riscv: Add missing barriers in cpu_switchto.
Details in comments.

sparc: Add missing barriers in cpu_switchto.

sparc64: Add missing barriers in cpu_switchto.
Details in comments.

vax: Note where cpu_switchto needs barriers.

Not sure vax has ever had working MULTIPROCESSOR, though, and I'm not
even sure how to spell store-before-load barriers on VAX, so no
functional change for now.

arm32: Optimization: Omit needless membar when triggering softint.

When we are triggering a softint, it can't already hold any mutexes.
So any path to mutex_exit(mtx) must go via mutex_enter(mtx), which is
always done with atomic r/m/w, and we need not issue any explicit
barrier between ci->ci_curlwp = softlwp and a potential load of
mtx->mtx_owner in mutex_exit.

PR kern/57240

XXX pullup-8
XXX pullup-9
XXX pullup-10

arm32: Add missing barriers in cpu_switchto.

Details in comments.

PR kern/57240

XXX pullup-8
XXX pullup-9
XXX pullup-10

Sync w/ HEAD.

sync with head

typo in comment

Sync w/ HEAD.

Ensure that r5 contains curlwp before DO_AST_AND_RESTORE_ALIGNMENT_FAULTS
in lwp_trampoline as required by the move to make ASTs operate per-LWP
rather than per-CPU.

Thanks to martin@ for bisecting the amap corruption he was seeing and
testing this fix.

#ifdef _ARM_ARCH_7 the dmbs

Mirror the changes to aarch64 and

- Switch to TPIDRPRW_IS_CURLWP, because curlwp is accessed much more often
  by MI code.  It also makes curlwp preemption safe,

- Make ASTs operate per-LWP rather than per-CPU, otherwise sometimes LWPs
  can see spurious ASTs (which doesn't cause a problem, it just means some
  time may be wasted).

- Make sure ASTs are always set on the same CPU as the target LWP, and
  delivered via IPI if posted from a remote CPU so that they are resolved
  quickly.

- Add some cache line padding to struct cpu_info.

- Add a memory barrier in a couple of places where ci_curlwp is set.  This
  is needed whenever an LWP that is resuming on the CPU could hold an
  adaptive mutex.  The barrier needs to drain the CPU's store buffer, so
  that the update to ci_curlwp becomes globally visible before the LWP can
  resume and call mutex_exit().

Add support for KASAN on ARMv[67]

Thanks to maxv for many pointers and reviews.

Whitespace

KNF (sort #includes)

Mostly merge changes from HEAD upto 20200411

Merge changes from current as of 20200406

Sync with head.

G/C

Sync with head.

oldlwp is always non-NULL in cpu_switchto so remove the test for NULL.

Hopefully fix some problems seen with MP support on non-x86, in particular
where curcpu() is defined as curlwp->l_cpu:

- mi_switch(): undo the ~2007ish optimisation to unlock curlwp before
  calling cpu_switchto().  It's not safe to let other actors mess with the
  LWP (in particular l->l_cpu) while it's still context switching.  This
  removes l->l_ctxswtch.

- Move the LP_RUNNING flag into l->l_flag and rename to LW_RUNNING since
  it's now covered by the LWP's lock.

- Ditch lwp_exit_switchaway() and just call mi_switch() instead.  Everything
  is in cache anyway so it wasn't buying much by trying to avoid saving old
  state.  This means cpu_switchto() will never be called with prevlwp ==
  NULL.

- Remove some KERNEL_LOCK handling which hasn't been needed for years.

Explicitly annotate FPU requirements for LLVM MC.

When using GCC, this annotations change the global state, but there is
no push/pop functionality for .fpu to avoid this problem. The state is
local to each inline assembler block with LLVM MC.

Typo in comment

Sync with HEAD

Sync with HEAD, resolve a couple of conflicts

Typo in comment

update from HEAD

Sync with HEAD

Whitespace (align comments)

Trailing whitespace

Sync with HEAD

Make TPIDRPRW_IS_CURLWP work for MULTIPROCESSOR.
get curcpu() from new lwp.
don't set lwp l_cpu (already done).
Remove support for __HAVE_UNNESTED_INTRS
don't set curlwp until after we are done saving the oldlwp.
disable interrupts when setting curlwp/kernel stack pointer.
Overall, these changes simplify cpu_switchto even more.

Sync with HEAD

Pull up following revision(s) (requested by skrll in ticket #646):
	sys/arch/arm/arm32/genassym.cf: revision 1.70
	sys/arch/arm/arm32/cpuswitch.S: revision 1.86-1.89

Only set vfp & tpid registers and do ras lookups if new
lwp is not LW_SYSTEM.

There is no need to save/restore l_private in softint_switch now that
cpu_switchto is fixed

Rework register usage in cpu_switchto so curcpu() is preserved across
ras_lookup.  Only set vfp & tpid registers and do ras lookups if new lwp
is not LW_SYSTEM.  (tested on RPI and atf tests on BPI by skrll).

Fix register usage in softint_switch.  load / restore l_private across
softint_dispatch

Make sure to save the user thread point in softint_switch in case it was
set just before we got an interrupt.  Otherwise if the softint blocks, the
old value would be restored and change lost.

src is too big these days to tolerate superfluous apostrophes.  It's
"its", people!

Rebase to HEAD as of a few days ago.

Rebase.

Fix wrong instruction; mcr => mrc

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs.  ("Protocol error: too many arguments")

sync with head

ARM_MMU_EXTENDED support.

Move pmap_recent_user to ci->ci_pmap_lastuser and
pmap_previous_active_lwp to ci->ci_lastlwp.  Fix some comments.

Merge armv7 support from HEAD, specifically support for the BCM5301X
and BCM56340 evbarm kernels.

Replicate mcr with equivalent vms instruction.

For load/store double, name the second register explicitly.

sync with head

Move parts of cpu.h that are not needed by MI code in <arm/locore.h>
Don't include <machine/cpu.h> or <machine/frame.h>, use <arm/locore.h>
Use <arm/asm.h> instead of <machine/arm.h>

Move parts of cpu.h that are not needed by MI code in <arm/locore.h>
Don't include <machine/cpu.h> or <machine/frame.h>, use <arm/locore.h>
Use <arm/asm.h> instead of <machine/arm.h>

resync from head

Don't include <machine/param.h> since we should be getting that stuff from
"assym.h"

resync with head

sync with head

sync with (a bit old) head

Make sure to load the FPEXC context on context switch (if there a VFP) so
that the VFP state will be what the LWP expects.  (This isn't needed on
PPC or MIPS since their FPU/VEC state is reflected in the PSL/CPO_STATUS
which is handled automatically.)

Rename pcb_sp/PCB_SP to pcb_ksp/PCB_KSP so that ipsec.c will compile.

ARMFPE hasn't compiled since NetBSD 4.  Remove it.
Complete support for FPU_VFP.
fpregs now contains vfpreg.
XXX vfpreg only has space for 16 64-bit FP registers though VFPv3 and later
have 32 64-bit FP registers.

Merge improved arm support (especially Cortex) from HEAD
including OMAP and BCM53xx support.

Resync to 2012-11-19 00:00:00 UTC

Use ENTRY_NP for lwp_trampoline

sync with head

After calling lwp_startup, set fp to 0 to terminate call stack.

Need to do a GET_CURCPU(r4) before invoking DO_AST_AND_RESTORE_ALIGNMENT_FAULTS

blx reg is V5, not V4T

DO_AST_AND_RESTORE_ALIGNMENT_FAULTS needs AST_ALIGNMENT_FAULT_LOCALS

Fix typo.

Rename ARM options PROCESS_ID_IS_CUR{CPU,LWP} to TPIDRPRW_IS_CUR{CPU,LWP}
since TPIDRPRW is the cp15 register name.
Initialize it early in start along with CI_ARM_CPUID.
Remove other initializations.
We alays have ci_curlwp.
Enable TIPRPRW_IS_CURCPU in std.beagle.
[tested on a beaglboard (cortex-a8)]

small rototill.
pcb_flags is dead.  PCB_NOALIGNFLT is now in stored l_md.md_flags as
MDLWP_NOALIGNFLT.  This avoids a few loads of the PCB in exception handling.
pcb_tf has been moved to l_md.md_tf.  Again this avoids a lot of pcb
references just to access or set this.  It also means that pcb doesn't
need to accessed by MI code.
Move pcb_onfault to after the pcb union.
Add pcb_sp macro to make code prettier.
Add lwp_settrapframe(l, tf) to set the l_md.md_tf field.
Use lwp_trapframe to access it (was process_frame but that name was changed
in a previous commit).
Kill off curpcb in acorn26.
Kill the checks for curlwp being NULL.
Move TRAP_USERMODE from arm32/fault.c to frame.h and a __PROG26 version.
Replace tests for usermode with that macro.

Kill curpcb/ci_curpcb.  Use device_t in cpu_info.
Add ci_softc (where ci_curpcb was so cpu_info doesn't change).

Rework VFP support to use PCU.
Add emulation of instruction which save/restore the VFP FPSCR.
Add a sysarch hook to VFP FPSCR manipulation.

[The emulation will be used by libc to store/fetch exception modes and
rounding mode on a per-thread basis.]

Sync with HEAD.

sync with head

Fetch user read-only thread and process id from l->l_private, not the pcb.
(need to g/c the pcb field formerly used for this).

sync with head

Sync with HEAD

include "assym.h" instead of pte.h

Retire struct user, remove sys/user.h inclusions.  Note sys/user.h header
as obsolete.  Remove USER_TO_UAREA/UAREA_TO_USER macros.

Various #include fixes and review by matt@.

sync with head

Rename L_ADDR to L_PCB and amend some comments accordingly.

sync with head.

Sync with HEAD.

Sync with HEAD.

Update haad-dm branch to haad-dm-base2.

Use IF32_bits instead of I32_bit.  Only disable/irq is __HAVE_UNNESTED_IRQS
is undefined.

Sync with HEAD.

sync with head.

sync with head.

Merge kernel changes in matt-armv6 to HEAD.

There's really no need to switch VM contexts within cpu_switchto() as
MI code always calls pmap_deactivate/pmap_activate on context switch.

Instead, just record the last active lwp (or NULL if it exited) and
defer switching VM context to pmap_activate(). This saves an additional
function call overhead in cpu_switchto().

While here, g/c unused cpuswitch.S local .Lblock_userspace_access.

Sync with HEAD.

sync with head.

sync with HEAD

Sync with head.

sync with head.

VFP support.

Sync with HEAD.

Sync with HEAD.

Add fast softint switch support.

sync with head

Sync with HEAD

Sync to HEAD.

Optimize cpu_switchto to store the new PCB address in r7, rather than
loading it from memory in 3 places.

Also adjust ordering of a few loads to try and avoid stalling.

With the removal of IPKDB on arm, the undefined stack is only used to
bounce into SVC32 mode, there is no per-process data stored on it.

We can therefore use the undefined stack setup by the platform machdep.c
as a system wide undefined stack.

This removes the need for a per-process undefined stack, and the processor
mode switching overhead it causes in cpu_switchto.

The space freed in the USPACE is used to increase the per process kernel
stack size.

Sync with HEAD

Take a micro-optimization from FreeBSD/arm.

When switching from SVC32->UND32 to read/write R13_und we don't need to clear
the mode bits as:
PSR_SVC32_MODE | PSR_UND32_MODE = PSR_UND32_MODE

While reading the code I also noted that interrupts are enabled for most of
the function as pmap_switch returns with interrupts in the state they are on
entry.  This appears to be different to what the code after pmap_switch
expects, in that the behaviour suggests they should be disabled.

Because of this I've made the writing of R13_und explicitly disable.
interupts as part of the mode switch.

This also means that the IRQenableALL call is now redundant as the
interrupts are already enabled.

XXX: it's not clear if arm_fpe_core_changecontext should be called with
interrupts disabled.

Remove unused items: IRQdisableALL, IRQenableALL & Lcpufuncs.

Tested on cats. lmbench shows no performance change.

Add and fix a couple of comments.

Push a switchframe in dumpsys and cpu_switchto, but as dumpsys calls
other funcs a switchframe needs to be a multiple of 8 bytes. Stash sp as
well in the switchframe to bump it to 24bytes.

Setup the switchframe appropriately in cpu_lwp_fork.

Remove savectx - nothing uses it.

All of this make gdb's life much easier when dealing with crash dumps and
live kernels.

Reviewd by chris.

Sync with HEAD.

Add AT91 support from Sami Kantoluoto
Add TI OMAP2430 support from Marty Fouts @ Danger Inc

Make sure instructions are aligned after .asciz

sync with HEAD

Sync with HEAD.

sync with head.

Merge the ppcoea-renovation branch to HEAD.

This branch was a major cleanup and rototill of many of the various OEA
cpu based PPC ports that focused on sharing as much code as possible
between the various ports to eliminate near-identical copies of files in
every tree.  Additionally there is a new PIC system that unifies the
interface to interrupt code for all different OEA ppc arches.  The work
for this branch was done by a variety of people, too long to list here.

TODO:
bebox still needs work to complete the transition to -renovation.
ofppc still needs a bunch of work, which I will be looking at.
ev64260 still needs to be renovated
amigappc was not attempted.

NOTES:
pmppc was removed as an arch, and moved to a evbppc target.

Sync with head.

Sync with HEAD

Sync with HEAD.

ARM cpu_switchto() has been partially broken since yamt-idlelwp was merged
as its cache/tlb management smarts relied too heavily on pre-merge context-
switch behaviour. See PR kern/36548 for one manifestation of the breakage.

To address this:
 - Ditch the shadow pmap variables in the PCB (pagedir, l1vec, dacr, cstate)
   as it was too easy for them to get out of sync with the pmap.
 - Re-write (and fix) the convoluted cpuswitch.S cache/tlb ASM code in C.
   It's only slightly less efficient, but is much more readable/maintainable.
 - Document cpufuncs.cf_context_switch() as being C-callable.
 - pmap_activate() becomes a no-op if the lwp's vmspace is already active.
   (Good performance win, since pmap_activate() is now invoked on every
   context-switch, even though ARM's cpu_switchto() already does all the
   grunt work)

XXX: Some CPU-specific armXX_context_switch() implementations (arm67,
arm7tdmi, arm8) always flush the I+D caches. This should not be necessary.
Someone with access to hardware (acorn32?) needs to deal with this.

Add a diagnostic check for cpu_switchto(x, NULL);

sync with head.

Reworked cpuswitch for armv6 and new world order

Sync with head.

Sync with HEAD.

Sync with head.

No need to check if oldl == newl in cpu_switchto. All the callers ensure
this is never the case.

Fixup a few comments while I'm here.

Update to HEAD.

merge yamt-idlelwp branch.  asked by core@.  some ports still needs work.

from doc/BRANCHES:

	idle lwp, and some changes depending on it.

	1. separate context switching and thread scheduling.
	   (cf. gmcgarry_ctxsw)
	2. implement idle lwp.
	3. clean up related MD/MI interfaces.
	4. make scheduler(s) modular.

Set curlwp in cpu_switchto and provide a cpu_did_resched.

Who is copying who now?

Adapt arm32. Thanks to scw for helping out.

Tested on my cats (SA1)

XXX hydra should die. i've made some changes, but no guarantees.

Pull up following revision(s) (requested by matt in ticket #470):
	sys/arch/evbarm/iq80310/iq80310_timer.c: revision 1.20
	sys/arch/evbarm/ifpga/pl030_rtc.c: revision 1.8
	sys/arch/evbarm/include/types.h: revision 1.7
	sys/arch/arm/arm32/genassym.cf: revision 1.30
	sys/arch/arm/arm32/cpuswitch.S: revision 1.46
Get DOMAIN_CLIENT directly from arm/arm32/pte.h instead of from genassym
to avoid redefinition when both assymh and pte.h are included (as in
INTEGRATOR's intmmu.S, which uses more macros from pte.h).
Convert evbarm to __HAVE_GENERIC_TODR.

- sync with head.
- move sched_changepri back to kern_synch.c as it doesn't know PPQ anymore.

sync with head.

Get DOMAIN_CLIENT directly from arm/arm32/pte.h instead of from genassym
to avoid redefinition when both assymh and pte.h are included (as in
INTEGRATOR's intmmu.S, which uses more macros from pte.h).

Merge newlock2 to head.

For now always call sched_unlock_idle/sched_lock_idle. They will be
removed by yamt's cpu_switchto() changes.

sync with head

sync with head.

Sync with head.

Merge 2006-05-24 NetBSD-current into the "peter-altq" branch.

sync with head.

Fix some more comments.

sync with head

Fix some comments.

merge ktrace-lwp.

Fix the sync with head I botched.

Sync with HEAD.

Sync with HEAD

 - Add LOCKDEBUG-protected calls to sched_lock_idle() to cpu_switchto and
   the idle loop. They seem to have gone AWOL sometime in the past.
   Fixes port-arm/23390.
 - While here, tidy up the idle loop.
 - Add a cheap DIAGNOSTIC check for run queue sanity.

Fix a braino introduced in r1.37. Thanks to Tom Spindler for spotting it.

Remove p_nras from struct proc - use LIST_EMPTY(&p->p_raslist) instead.
Remove p_raslock and rename p_lwplock p_lock (one lock is enough).
Simplify window test when adding a ras and correct test on VM_MAXUSER_ADDRESS.
Avoid unpredictable branch in i386 locore.S
(pad fields left in struct proc to avoid kernel bump)

Don't drop to spl0 in cpu_switch/cpu_switchto. Do it in the idle loop
instead.

With this change, we no longer need to save the current interrupt level
in the switchframe. This is no great loss since both cpu_switch and
cpu_switchto are always called at splsched, so the process' spl is
effectively saved somewhere in the callstack.

This fixes an evbarm problem reported by Allen Briggs:

        lwp gets into sa_switch -> mi_switch with newl != NULL
            when it's the last element on the runqueue, so it
            hits the second bit of:
                if (newl == NULL) {
                        retval = cpu_switch(l, NULL);
                } else {
                        remrunqueue(newl);
                        cpu_switchto(l, newl);
                        retval = 0;
                }

        mi_switch calls remrunqueue() and cpu_switchto()

        cpu_switchto unlocks the sched lock
        cpu_switchto drops CPU priority
        softclock is received
        schedcpu is called from softclock
        schedcpu hits the first if () {} block here:
                if (l->l_priority >= PUSER) {
                        if (l->l_stat == LSRUN &&
                            (l->l_flag & L_INMEM) &&
                            (l->l_priority / PPQ) != (l->l_usrpri / PPQ)) {
                                remrunqueue(l);
                                l->l_priority = l->l_usrpri;
                                setrunqueue(l);
                        } else
                                l->l_priority = l->l_usrpri;
                }

        Since mi_switch has already run remrunqueue, the LWP has been
            removed, but it's not been put back on any queue, so the
            remrunqueue panics.

A couple of Xscale tweaks:

 - Use the "clz" instruction to pick a run-queue, instead of using the
   ffs-by-table-lookup method.
 - Use strd instead of stmia where possible.
 - Use multiple ldr instructions instead of ldmia where possible.

Make sure to include opt_foo.h if a defflag option FOO is used.

Fix for port-arm/21962.  Rather than fixing the #ifndef spl0, I removed
the test as spl0 is actually a macro for splx(0). The code now calls
splx(0)

(note building with the #ifdef fixed, caused the build to fail on a
GENERIC acorn32 kernel.)

Fix LINTSTUB comments.

Remove #ifdefs supporting the old pmap, switching fully to the new.

Remove a strh.  I don't think it's available on archv3 and it doesn't work
on acorn32's with an SA110 in them as the bus doesn't support halfword
transfers.

Some ARM32_PMAP_NEW-related cleanup:
* Define a new "MMU type", ARM_MMU_SA1.  While the SA-1's MMU is basically
  compatible with the generic, the SA-1 cache does not have a write-through
  mode, and it is useful to know have an indication of this.
* Add a new PMAP_NEEDS_PTE_SYNC indicator, and try to evaluate it at
  compile time.  We evaluate it like so:
  - If SA-1-style MMU is the only type configured -> 1
  - If SA-1-style MMU is not configured -> 0
  - Otherwise, defer to a run-time variable.
  If PMAP_NEEDS_PTE_SYNC might evaluate to true (SA-1 only or run-time
  check), then we also define PMAP_INCLUDE_PTE_SYNC so that e.g. assembly
  code can include the necessary run-time support.  PMAP_INCLUDE_PTE_SYNC
  largely replaces the ARM32_PMAP_NEEDS_PTE_SYNC manual setting Steve
  included with the original new pmap.
* In the new pmap, make pmap_pte_init_generic() check to see if the CPU
  has a write-back cache.  If so, init the PT cache mode to C=1,B=0 to get
  write-through mode.  Otherwise, init the PT cache mode to C=1,B=1.
* Add a new pmap_pte_init_arm8().  Old pmap, same as generic.  New pmap,
  sets page table cacheability to 0 (ARM8 has a write-back cache, but
  flushing it is quite expensive).
* In the new pmap, make pmap_pte_init_arm9() reset the PT cache mode to
  C=1,B=0, since the write-back check in generic gets it wrong for ARM9,
  since we use write-through mode all the time on ARM9 right now.  (What
  this really tells me is that the test for write-through cache is less
  than perfect, but we can fix that later.)
* Add a new pmap_pte_init_sa1().  Old pmap, same as generic.  New pmap,
  does generic initialization, then resets page table cache mode to
  C=1,B=1, since C=1,B=0 does not produce write-through on the SA-1.

Add the generic arm32 bits of the new pmap, contributed by Wasabi Systems.

Some features of the new pmap are:

 - It allows L1 descriptor tables to be shared efficiently between
   multiple processes. A typical "maxusers 32" kernel, where NPROC is set
   to 532, requires 35 L1s. A "maxusers 2" kernel runs quite happily
   with just 4 L1s. This completely solves the problem of running out
   of contiguous physical memory for allocating new L1s at runtime on a
   busy system.

 - Much improved cache/TLB management "smarts". This change ripples
   out to encompass the low-level context switch code, which is also
   much smarter about when to flush the cache/TLB, and when not to.

 - Faster allocation of L2 page tables and associated metadata thanks,
   in part, to the pool_cache enhancements recently contributed to
   NetBSD by Wasabi Systems.

 - Faster VM space teardown due to accurate referenced tracking of L2
   page tables.

 - Better/faster cache-alias tracking.

The new pmap is enabled by adding options ARM32_PMAP_NEW to the kernel
config file, and making the necessary changes to the port-specific
initarm() function. Several ports have already been converted and will
be committed shortly.

Merge the nathanw_sa branch.

Shave a couple of cycles off the beginning of cpu_switch().

Merge switch_exit() and switch_lwp_exit(), and hand-schedule the
resulting function to avoid stalls on StrongARM and XScale.

Rename cpu_preempt() to cpu_switchto(), and make the caller remove the
new process from its run queue before calling cpu_switchto().

While here, make a few cpu_switch() and cpu_switchto() implementations
get the outgoing LWP from the args, rather than looking at the curlwp
variable.

Flush the cache before reading sched_whichqs.  This is entirely the wrong
way to do this, but it should work (very slowly) until I work out the
right way.

Switch to the idle PCB in the idle loop.

In cpu_switch and cpu_exit, use curcpu to find curproc and curpcb, rather
than assuming CPU 0.  Also fix a register-shuffling botch in cpu_exit.

Reshuffle register usage in cpu_exit along the lines of cpu_switch to
reduce saving and restoring of registers.

Redo the following revision:
syssrc/sys/arch/arm/arm32/cpuswitch.S		1.27

Original log message:

The grand cpu_switch register reshuffle!

In particular, use r8 to hold the old process, and r7 for medium-term
scratch, saving r0-r3 for things we don't need saved over function
calls.  This gets rid of five register-to-register MOVs.

Re-do the following revisions, this time on a branch where they won't
interfere with the nathanw_sa merge:

syssrc/sys/arch/arm/arm32/cpuswitch.S			1.26
syssrc/sys/arch/arm/arm32/genassym.cf			1.18
syssrc/sys/arch/arm/arm32/vm_machdep.c			1.21
syssrc/sys/arch/arm/include/pcb.h			1.5

Original commit message:

In cpu_switch(), stack more registers at the start of the function,
and hence save fewer into the PCB.  This should give me enough free
registers in cpu_switch to tidy things up and support MULTIPROCESSOR
properly.  While we're here, make the stacked registers into an
APCS stack frame, so that DDB backtraces through cpu_switch() will
work.

This also affects cpu_fork(), which has to fabricate a switchframe and
PCB for the new process.

file cpuswitch.S was added on branch bjh21-hydra on 2002-10-19 11:59:36 +0000

Undo recent cpu_switch register usage changes in order to decrease nathanw_sa
merge pain.

The grand cpu_switch register reshuffle!

In particular, use r8 to hold the old process, and r7 for medium-term
scratch, saving r0-r3 for things we don't need saved over function
calls.  This gets rid of five register-to-register MOVs.

In cpu_switch(), stack more registers at the start of the function,
and hence save fewer into the PCB.  This should give me enough free
registers in cpu_switch to tidy things up and support MULTIPROCESSOR
properly.  While we're here, make the stacked registers into an
APCS stack frame, so that DDB backtraces through cpu_switch() will
work.

This also affects cpu_fork(), which has to fabricate a switchframe and
PCB for the new process.

Catch up to -current.

Switch to using the MI C versions of setrunqueue() and remrunqueue().
GCC produces almost exactly the same instructions as the hand-assembled
versions, albeit in a different order.  It even found one place where it
could shave one off.  Its insistence on creating a stack frame might slow
things down marginally, but not, I think, enough to matter.

Continue the " - . - 8" purge.  Specifically:

add	rd, pc, #foo - . - 8		->	adr	rd, foo
ldr	rd, [pc, #foo - . - 8]		->	ldr	rd, foo

Also, when saving the return address for a function pointer call, use
"mov lr, pc" just before the call unless the return address is somewhere
other than just after the call site.

Finally, a few obvious little micro-optimisations like using LDR directly
rather than ADR followed by LDR, and loading directly into PC rather than
bouncing via R0.

Instead of "add rd, pc, #foo - . - 8", use either "adr rd, foo" or (where
appropriate) "mov lr, pc".  This makes things slightly less confusing and
ugly.

Move curpcb into struct cpu_info in MULTIPROCESSOR kernels.

Use ADR rather than an explicit ADD from PC.

Remove an outdated register assignment comment.

Minimal changes to allow a kernel with "options MULTIPROCESSOR" to compile
and boot multi-user on a single-processor machine.  Many of these changes
are wildly inappropriate for actual multi-processor operation, and correcting
this will be my next task.

Fix cpu_switch() after RAS integration from trunk.

Rearrange slightly and pass p, not l, to ras_lookup().

Catch up to -current.

sync kqueue branch with HEAD

catch up with -current.

Add machine-dependent bits of RAS for arm32.

catch up with -current.

Check to see if the incoming LWP has the same L1 table as the
outgoing LWP.  If so, then we can skip the cache purge and TTB
reload.  This results in a ~40% reduction in cache purges called
from cpu_switch() in my test using two threaded applications which
communicate with each other.

Partial (ARM only) sync with trunk -- significant performance improvements
for XScale-based systems.

More local label fixups.

Must ... micro ... optimize!

* Save an instruction in the transition from idle to have-process-to-
  switch-to, and eliminate two instructions that cause datadep-stalls
  on StrongARM And XScale (one in each idle block).
* Rearrange some other instructions to avoid datadep-stalls on StrongARM
  and XScale.
* Since cpu_do_powersave == 0 is by far the common case, avoid a
  pipeline flush by reordering the two idle blocks.

* Add a new machdep.powersave sysctl, which controls the use of
  the CPU's "sleep" function in the idle loop.
* Default all CPUs to not use powersave, except for the PDA processors
  (SA11x0 and PXA2x0).

This significantly reduces inteterrupt latency in high-performance
applications (and was good to squeeze another ~10% out of an XScale
IOP on a Gig-E benchmark).

 * Use local label names (.Lfoo vs. (Lfoo or foo))
 * When moving from cpsr, use "cpsr" instead of "cpsr_all" (which is
   provided, but doesn't make sense since mrs doesn't support fields
   like msr does).

We only need to modify the CPSR's control field, so use cpsr_c rather
than cpsr_all.

Tweak asm to avoid a couple of stalls.

Rearrange the beginning of cpu_switch() slightly to reduce data-dep
stalls on StrongARM and XScale.

Rearrange the beginning of cpu_switch() slightly to reduce data-dep
stalls on StrongARM and XScale.

Add the requisite calls to sched_lock_idle() and sched_unlock_idle() if
LOCKDEBUG is defined, as is done on the trunk.

More register usage tweaks to reduce differences with trunk.

Tweak register usage in cpu_switch() slightly to reduce differences
with the trunk.

Tweak register usage in cpu_preempt() slightly.

Reduce some differences with the trunk.

Make a slight tweak to register usage to save an instruction.

Rearrange the exit path so that we don't do a idcache_wbinv_all *twice*
when a process or lwp exits.

Rearrange the exit path so that we don't do a idcache_wbinv_all *twice*
when a process exits.

* In switch_exit()/switch_lwp_exit(), set the outgoing-lwp register to
  NULL (rather than lwp0) so that we actually use the "exiting process"
  optimization in cpu_switch().
* Correct some comments.

* Pass proc0 to switch_exit(), to make this a little more like the
  nathanw_sa branch.
* In switch_exit(), set the outgoing-proc register to NULL (rather than
  proc0) so that we actually use the "exiting process" optimization in
  cpu_switch().

Fix cpu_preempt() for the __NEWINTR case.

Back out the changes that implement the scheduler locking protocol.
The register usage in this file is very different than on the trunk,
and so the changes made to the trunk don't really apply here.

Fix up some comments while here.

Curproc->curlwp renaming.

Change uses of "curproc->l_proc" back to "curproc", which is more like the
original use. Bare uses of "curproc" are now "curlwp".

"curproc" is now #defined in proc.h as ((curlwp) ? (curlwp)->l_proc) : NULL)
so that it is always safe to reference curproc (*de*referencing curproc
is another story, but that's always been true).

catch up with -current on kqueue branch

switch_search -> Lswitch_search in switch_exit().

Catch up to -current.

Implement scheduler lock protocol, this fixes PR arm/10863.

Also add correct locking when freeing pages in pmap_destroy (fix from potr)

This now means that arm32 kernels can be built with LOCKDEBUG enabled. (only tested on cats though)

Make this compile again.

Catch up to -current.

Sync w/ -current.

Overhaul of the ARM cache code.  This is mostly a simplification
pass.  Rather than providing a whole slew of cache operations that
aren't ever used, distill them down to some useful primitives:

	icache_sync_all         Synchronize I-cache
	icache_sync_range       Synchronize I-cache range

	dcache_wbinv_all        Write-back and Invalidate D-cache
	dcache_wbinv_range      Write-back and Invalidate D-cache range
	dcache_inv_range        Invalidate D-cache range
	dcache_wb_range         Write-back D-cache range

	idcache_wbinv_all       Write-back and Invalidate D-cache,
				Invalidate I-cache
	idcache_wbinv_range     Write-back and Invalidate D-cache,
				Invalidate I-cache range

Note: This does not yet include an overhaul of the actual asm files
that implement the primitives.  Instead, we've provided a safe default
for each CPU type, and the individual CPU types can now be optimized
one at a time.

Sync kqueue branch with -current.

Catch up to -current.

Provide a way for platforms to move away from the old RiscPC-centric
interrupt code.  Garbage-collect some unused stuff.

l_stat is an int now instead of a u_char.  Pointed out by pk in a
commit message.

Give the idle loop a non-profiled entry, means it appears in profile info correctly (rather than all it's time being under remrunqueue)
switch_exit only needs to take 1 parameter, it loads the value of proc0 into R1 itself
Fixup some comments to reflect the real state of things.
Tweak a couple of bits of asm to avoid a load delay.
remove excess code for setting curpcb and curproc.

Machine-dependent kernel mods for scheduler activations on
32-bit ARM processors.  Kernel boots multi-user on an XScale,
but upcalls not yet tested.

Sync the thorpej-mips-cache branch with -current.

file cpuswitch.S was added on branch nathanw_sa on 2001-11-15 06:39:21 +0000

Correct comments for ffs algoritm (it isn't using register r0)

Catch up with -current.

Fix .type which uses wrong symbol name.

update to -current

file cpuswitch.S was added on branch kqueue on 2001-08-03 04:10:58 +0000

Move the generic arm32 files into arm/arm32 from arm32/arm32, tested kernel builds on cats and riscpc.