CVS log for src/sys/uvm/uvm_amap.c

Up to [cvs.NetBSD.org] / src / sys / uvm

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

Align uvm_amap to COHERENCY_UNIT.

s/value value/value/ in comment. No functional change.

uvm(9): KASSERT(A && B) -> KASSERT(A); KASSERT(B)

Sync with HEAD.

Consistently use %#jx instead of 0x%jx or just %jx in UVMHIST_LOG formats

Pull up following revision(s) (requested by chs in ticket #1095):

	sys/uvm/uvm_amap.c: revision 1.124 (via patch)
	sys/uvm/uvm_amap.c: revision 1.125 (via patch)
	sys/uvm/uvm_io.c: revision 1.29 (via patch)

Effectively disable the AMAP_REFALL flag because it is unsafe.

This flag tells the amap code that it does not need to allocate ppref
as part of adding or removing a reference, but that is only correct
if the range of the reference being added or removed is the same
as the range of all other references to the amap, and the point of
this flag is exactly to try to optimize the case where the range is
different and thus this flag would not be correct to use.
Fixes PR 55366.

The previous fix for PR 55366 in uvm_amap.c 1.124 was incomplete:
 - amap_adjref_anons() must also ignore AMAP_REFALL when updating
   the ppref, not just when deciding whether or not to initialize ppref.
 - UVM_EXTRACT_QREF relies on AMAP_REFALL to work properly,
   and since we can't use AMAP_REFALL then we can't use QREF either.

the previous fix for PR 55366 in uvm_amap.c 1.124 was incomplete:
 - amap_adjref_anons() must also ignore AMAP_REFALL when updating
   the ppref, not just when deciding whether or not to initialize ppref.
 - UVM_EXTRACT_QREF relies on AMAP_REFALL to work properly,
   and since we can't use AMAP_REFALL then we can't use QREF either.

Effectively disable the AMAP_REFALL flag because it is unsafe.
This flag tells the amap code that it does not need to allocate ppref
as part of adding or removing a reference, but that is only correct
if the range of the reference being added or removed is the same
as the range of all other references to the amap, and the point of
this flag is exactly to try to optimize the case where the range is
different and thus this flag would not be correct to use.
Fixes PR 55366.

Pull up following revision(s) (requested by chs in ticket #1598):

	sys/uvm/uvm_amap.c: revision 1.123 (via patch)

fix amap_extend() to handle amaps where we previously failed to allocate
the ppref memory.

Pull up following revision(s) (requested by chs in ticket #1057):

	sys/uvm/uvm_amap.c: revision 1.123 (via patch)

fix amap_extend() to handle amaps where we previously failed to allocate
the ppref memory.

fix amap_extend() to handle amaps where we previously failed to allocate
the ppref memory.

Consistently use UVMHIST(__func__)

Convert UVMHIST_{CALLED,LOG} into UVMHIST_CALLARGS

Trailing whitespace

Mark amappl with PR_LARGECACHE.

Merge changes from current as of 20200406

Go back to freeing struct vm_anon one by one.  There may have been an
advantage circa ~2008 but there isn't now.

Make page waits (WANTED vs BUSY) interlocked by pg->interlock.  Gets RW
locks out of the equation for sleep/wakeup, and allows observing+waiting
for busy pages when holding only a read lock.  Proposed on tech-kern.

- Hide the details of SPCF_SHOULDYIELD and related behind a couple of small
  functions: preempt_point() and preempt_needed().

- preempt(): if the LWP has exceeded its timeslice in kernel, strip it of
  any priority boost gained earlier from blocking.

Sync with head.

0x%#x --> %#x for non-external codes.
Also, stop mixing up 0x%x and %#x in single files as far as possible.

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart.  v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap.  Others to follow later.

Back out the amap allocation  changes from earlier today - have seen a panic
with them.  Retain the lock changes.

- Start trying to reduce the high cache miss rate observed around vm_amap.
  On _LP64, pad struct vm_amap to 128 bytes and use the additional space to
  hold the arrays for tiny amaps which are common.  Carefully size the array
  allocations to avoid false sharing, and for smaller amaps try to share
  allocated cache lines.

- Eliminate most contention due to amap_list: maintain the list in the pool
  cache constructor / destructor like we do for struct file.  Cache the
  mutexes we allocate here.

- Don't do PR_WAITOK mutex allocations when NOWAIT has been specified.

PR kern/54821: 9.99.32 assertion in uvm_pageactivate

Looks like I forgot to commit this file yesterday.

Break the global uvm_pageqlock into a per-page identity lock and a private
lock for use of the pagedaemon policy code.  Discussed on tech-kern.

PR kern/54209: NetBSD 8 large memory performance extremely low
PR kern/54210: NetBSD-8 processes presumably not exiting
PR kern/54727: writing a large file causes unreasonable system behaviour

Activate pages in batch instead of acquring uvm_pageqlock a zillion times.

Sync with HEAD

Sync with HEAD

Resolve a couple of conflicts (result of the uimin/uimax changes)

Rename 'slotspace' -> 'slotarea' in UVM, to avoid (future) collision with
the x86 slotspace structure.

update from HEAD

Pull up following revision(s) (requested by pgoyette in ticket #335):
	share/man/man9/kernhist.9: 1.5-1.8
	sys/arch/acorn26/acorn26/pmap.c: 1.39
	sys/arch/arm/arm32/fault.c: 1.105 via patch
	sys/arch/arm/arm32/pmap.c: 1.350, 1.359
	sys/arch/arm/broadcom/bcm2835_bsc.c: 1.7
	sys/arch/arm/omap/if_cpsw.c: 1.20
	sys/arch/arm/omap/tiotg.c: 1.7
	sys/arch/evbarm/conf/RPI2_INSTALL: 1.3
	sys/dev/ic/sl811hs.c: 1.98
	sys/dev/usb/ehci.c: 1.256
	sys/dev/usb/if_axe.c: 1.83
	sys/dev/usb/motg.c: 1.18
	sys/dev/usb/ohci.c: 1.274
	sys/dev/usb/ucom.c: 1.119
	sys/dev/usb/uhci.c: 1.277
	sys/dev/usb/uhub.c: 1.137
	sys/dev/usb/umass.c: 1.160-1.162
	sys/dev/usb/umass_quirks.c: 1.100
	sys/dev/usb/umass_scsipi.c: 1.55
	sys/dev/usb/usb.c: 1.168
	sys/dev/usb/usb_mem.c: 1.70
	sys/dev/usb/usb_subr.c: 1.221
	sys/dev/usb/usbdi.c: 1.175
	sys/dev/usb/usbdi_util.c: 1.67-1.70
	sys/dev/usb/usbroothub.c: 1.3
	sys/dev/usb/xhci.c: 1.75
	sys/external/bsd/drm2/dist/drm/i915/i915_gem.c: 1.34
	sys/kern/kern_history.c: 1.15
	sys/kern/kern_xxx.c: 1.74
	sys/kern/vfs_bio.c: 1.275-1.276
	sys/miscfs/genfs/genfs_io.c: 1.71
	sys/sys/kernhist.h: 1.21
	sys/ufs/ffs/ffs_balloc.c: 1.63
	sys/ufs/lfs/lfs_vfsops.c: 1.361
	sys/ufs/lfs/ulfs_inode.c: 1.21
	sys/ufs/lfs/ulfs_vnops.c: 1.52
	sys/ufs/ufs/ufs_inode.c: 1.102
	sys/ufs/ufs/ufs_vnops.c: 1.239
	sys/uvm/pmap/pmap.c: 1.37-1.39
	sys/uvm/pmap/pmap_tlb.c: 1.22
	sys/uvm/uvm_amap.c: 1.108
	sys/uvm/uvm_anon.c: 1.64
	sys/uvm/uvm_aobj.c: 1.126
	sys/uvm/uvm_bio.c: 1.91
	sys/uvm/uvm_device.c: 1.66
	sys/uvm/uvm_fault.c: 1.201
	sys/uvm/uvm_km.c: 1.144
	sys/uvm/uvm_loan.c: 1.85
	sys/uvm/uvm_map.c: 1.353
	sys/uvm/uvm_page.c: 1.194
	sys/uvm/uvm_pager.c: 1.111
	sys/uvm/uvm_pdaemon.c: 1.109
	sys/uvm/uvm_swap.c: 1.175
	sys/uvm/uvm_vnode.c: 1.103
	usr.bin/vmstat/vmstat.c: 1.219
Reorder to test for null before null deref in debug code
--
Reorder to test for null before null deref in debug code
--
KNF
--
No need for '\n' in UVMHIST_LOG
--
normalise a BIOHIST log message
--
Update the kernhist(9) kernel history code to address issues identified
in PR kern/52639, as well as some general cleaning-up...
(As proposed on tech-kern@ with additional changes and enhancements.)
Details of changes:
* All history arguments are now stored as uintmax_t values[1], both in
   the kernel and in the structures used for exporting the history data
   to userland via sysctl(9).  This avoids problems on some architectures
   where passing a 64-bit (or larger) value to printf(3) can cause it to
   process the value as multiple arguments.  (This can be particularly
   problematic when printf()'s format string is not a literal, since in
   that case the compiler cannot know how large each argument should be.)
* Update the data structures used for exporting kernel history data to
   include a version number as well as the length of history arguments.
* All [2] existing users of kernhist(9) have had their format strings
   updated.  Each format specifier now includes an explicit length
   modifier 'j' to refer to numeric values of the size of uintmax_t.
* All [2] existing users of kernhist(9) have had their format strings
   updated to replace uses of "%p" with "%#jx", and the pointer
   arguments are now cast to (uintptr_t) before being subsequently cast
   to (uintmax_t).  This is needed to avoid compiler warnings about
   casting "pointer to integer of a different size."
* All [2] existing users of kernhist(9) have had instances of "%s" or
   "%c" format strings replaced with numeric formats; several instances
   of mis-match between format string and argument list have been fixed.
* vmstat(1) has been modified to handle the new size of arguments in the
   history data as exported by sysctl(9).
* vmstat(1) now provides a warning message if the history requested with
   the -u option does not exist (previously, this condition was silently
   ignored, with only a single blank line being printed).
* vmstat(1) now checks the version and argument length included in the
   data exported via sysctl(9) and exits if they do not match the values
   with which vmstat was built.
* The kernhist(9) man-page has been updated to note the additional
   requirements imposed on the format strings, along with several other
   minor changes and enhancements.
[1] It would have been possible to use an explicit length (for example,
     uint64_t) for the history arguments.  But that would require another
     "rototill" of all the users in the future when we add support for an
     architecture that supports a larger size.  Also, the printf(3)
format
     specifiers for explicitly-sized values, such as "%"PRIu64, are much
     more verbose (and less aesthetically appealing, IMHO) than simply
     using "%ju".
[2] I've tried very hard to find "all [the] existing users of
kernhist(9)"
     but it is possible that I've missed some of them.  I would be glad
to
     update any stragglers that anyone identifies.
--
For some reason this single kernel seems to have outgrown its declared
size as a result of the kernhist(9) changes.  Bump the size.
XXX The amount of increase may be excessive - anyone with more detailed
XXX knowledge please feel free to further adjust the value
appropriately.
--
Misssed one cast of pointer --> uintptr_t in previous kernhist(9) commit
--
And yet another one.  :(
--
Use correct mark-up for NetBSD version.
--
More improvements in grammar and readability.
--
Remove a stray '"' (obvious typo) and add a couple of casts that are
probably needed.
--
And replace an instance of "%p" conversion with "%#jx"
--
Whitespace fix. Give Bl tag table a width. Fix Xr.

Update the kernhist(9) kernel history code to address issues identified
in PR kern/52639, as well as some general cleaning-up...

(As proposed on tech-kern@ with additional changes and enhancements.)

Details of changes:

* All history arguments are now stored as uintmax_t values[1], both in
  the kernel and in the structures used for exporting the history data
  to userland via sysctl(9).  This avoids problems on some architectures
  where passing a 64-bit (or larger) value to printf(3) can cause it to
  process the value as multiple arguments.  (This can be particularly
  problematic when printf()'s format string is not a literal, since in
  that case the compiler cannot know how large each argument should be.)

* Update the data structures used for exporting kernel history data to
  include a version number as well as the length of history arguments.

* All [2] existing users of kernhist(9) have had their format strings
  updated.  Each format specifier now includes an explicit length
  modifier 'j' to refer to numeric values of the size of uintmax_t.

* All [2] existing users of kernhist(9) have had their format strings
  updated to replace uses of "%p" with "%#jx", and the pointer
  arguments are now cast to (uintptr_t) before being subsequently cast
  to (uintmax_t).  This is needed to avoid compiler warnings about
  casting "pointer to integer of a different size."

* All [2] existing users of kernhist(9) have had instances of "%s" or
  "%c" format strings replaced with numeric formats; several instances
  of mis-match between format string and argument list have been fixed.

* vmstat(1) has been modified to handle the new size of arguments in the
  history data as exported by sysctl(9).

* vmstat(1) now provides a warning message if the history requested with
  the -u option does not exist (previously, this condition was silently
  ignored, with only a single blank line being printed).

* vmstat(1) now checks the version and argument length included in the
  data exported via sysctl(9) and exits if they do not match the values
  with which vmstat was built.

* The kernhist(9) man-page has been updated to note the additional
  requirements imposed on the format strings, along with several other
  minor changes and enhancements.

[1] It would have been possible to use an explicit length (for example,
    uint64_t) for the history arguments.  But that would require another
    "rototill" of all the users in the future when we add support for an
    architecture that supports a larger size.  Also, the printf(3) format
    specifiers for explicitly-sized values, such as "%"PRIu64, are much
    more verbose (and less aesthetically appealing, IMHO) than simply
    using "%ju".

[2] I've tried very hard to find "all [the] existing users of kernhist(9)"
    but it is possible that I've missed some of them.  I would be glad to
    update any stragglers that anyone identifies.

sync to latest -current.

sync with head

Pull up following revision(s) (requested by chs in ticket #173):
	sys/uvm/uvm_amap.c: revision 1.107
initialize amap per-page reference counts before changing the amap's
overall reference count.  this fixes the crashes seen for the last 9 months
with web browers and plugins, which was also the cause of PR 46193.

initialize amap per-page reference counts before changing the amap's
overall reference count.  this fixes the crashes seen for the last 9 months
with web browers and plugins, which was also the cause of PR 46193.

sync to latest -current.

Pull up following revision(s) (requested by chs in ticket #150):
	sys/uvm/uvm_amap.c: revision 1.106
adjust amap_cow_now() to make UVM_PAGE_TRKOWN happy.

adjust amap_cow_now() to make UVM_PAGE_TRKOWN happy.

Improve UVM_PAGE_TRKOWN.
Add more asserts to uvm_page.

merge to -current.

Major changes to uvm.
Support multiple collections (groups) of free pages and run the page
reclaimation algorithm on each group independently.

extending vmem(9) to be able to allocated resources for it's own needs.
simplifying uvm_map handling (no special kernel entries anymore no relocking)
make malloc(9) a thin wrapper around kmem(9)
(with private interface for interrupt safety reasons)

releng@ acknowledged

- use O->A loan to serve read(2).  based on a patch from Chuck Silvers
- associated O->A loan fixes.

assertion

amap_cow_now: just free the fresh anon on error, no need to dispose it.

- Rework uvm_anfree() into uvm_anon_freelst(), which always drops the lock.
- Free anons in uvm_anon_freelst() without lock held.
- Mechanic sync to unused loaning code.

reduce the number of atomic ops in common cases.  it's exceptional for
anons to remain longer than amap.

amap_copy(): Keep the source amap locked until its lock has been copied.

Kernel assertion "anon->an_lock == amap->am_lock" no longer fails.

Ok: Mindaugas Rasiukevicius <rmind@netbsd.org>

amap_copy: fix one more regression, thanks to enami@.

Fix uvmplock regression - a lock against oneself case in amap_swap_off().
Happens since amap is NULL in uvmfault_anonget(), so uvmfault_unlockall()
keeps anon locked, when it should unlock it.

amap_pp_adjref: fix regression, spotted by nonaka@.

Clean-up, add asserts, slightly simplify.

Catchup with rmind-uvmplock merge.

Clean up, sprinkle asserts, consify, use unsigned, use kmem_zalloc instead
of memset, reduce the scope of some variables, improve some comments.

No functional change intended.

amap_add/amap_unadd: clean up slightly, use unsigned, add asserts.

Add amap_adjref_anons() helper and simplify amap_ref()/amap_unref().

amap_lookup{s}: add assert, clean-up slightly.

Welcome to 5.99.53!  Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
  New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
  the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
  Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
  kernel-lock on some ports).  Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

Sync with HEAD.

sync with head

Replace "malloc" in comments, remove unnecessary header inclusions.

sync with head

Sync with HEAD

udpate license clauses on my code to match the new-style BSD licenses.
based on diff that rmind@ sent me.

no functional change with this commit.

Reorganise UVM locking to protect P->V state and serialise pmap(9)
operations on the same page(s) by always locking their owner.  Hence
lock order: "vmpage"-lock -> pmap-lock.

Patch, proposed on tech-kern@, from Andrew Doran.

sync with head

Remove uarea swap-out functionality:

- Addresses the issue described in PR/38828.
- Some simplification in threading and sleepq subsystems.
- Eliminates pmap_collect() and, as a side note, allows pmap optimisations.
- Eliminates XS_CTL_DATA_ONSTACK in scsipi code.
- Avoids few scans on LWP list and thus potentially long holds of proc_lock.
- Cuts ~1.5k lines of code.  Reduces amd64 kernel size by ~4k.
- Removes __SWAP_BROKEN cases.

Tested on x86, mips, acorn32 (thanks <mpumford>) and partly tested on
acorn26 (thanks to <bjh21>).

Discussed on <tech-kern>, reviewed by <ad>.

sync with head.

assertions

Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.

sync with head.

Sync with HEAD.

Convert some panic() checks to KASSERT()s.
This code is stable and there is no reason to enforce checks.

Apply patch (requested by ad in ticket #357):
Make adjustment of some critical variables atomic.

Sync with HEAD.

Sync with HEAD.

Update haad-dm branch to haad-dm-base2.

Make adjustment of uvm_extrapages atomic since it's done without a lock.
XXX This is still a hack.

Sync with HEAD.

sync with head

sync with HEAD

Sync with HEAD

Merge vmlocking2 to head.

Sync with HEAD.

Pullup to HEAD

Sync with head.

Allocate amaps from a pool_cache.

Pull the vmlocking changes into a new branch.

sync with head.

Use pool_cache for allocating a few more types of objects.

Sync with HEAD.

Merge unobtrusive locking changes from the vmlocking branch.

file uvm_amap.c was added on branch matt-mips64 on 2007-07-21 19:21:54 +0000

Merge unobtrusive locking changes from the vmlocking branch.

Sync with head.

Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

- Put a per-LWP lock around swapin / swapout.
- Replace use of lockmgr().
- Minor locking fixes and assertions.
- uvm_map.h no longer pulls in proc.h, etc.
- Use kpause where appropriate.

sync with head.

Pull in the initial set of changes for the vmlocking branch.

Sync with head.

Pass an ipl argument to pool_init/POOL_INIT to be used when initializing
the pool's lock.

- sync with head.
- move sched_changepri back to kern_synch.c as it doesn't know PPQ anymore.

sync with head.

TRUE -> true, FALSE -> false

Replace the Mach-derived boolean_t type with the C99 bool type.  A
future commit will replace use of TRUE and FALSE with true and false.

Merge newlock2 to head.

Remove support for SA. Ok core@.

sync with head.

sync with head.

Sync with head.

remove some __unused from function parameters.

sync with head

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

sync with head

Merge from HEAD.

sync with head.

make amap use kmem_alloc, rather than malloc.
(ie. make it use kernel_map, rather than kmem_map.)
kmem_map is more restricted than kernel_map,
and there's no point for amap to use it.

sync with head.

Merge 2006-05-24 NetBSD-current into the "peter-altq" branch.

sync with head.

sync with head

Sync with head.

amap_splitref: assert that origref->ar_amap is initialized
by caller beforehand.

sync with head.

- amap_copy: take a "flags" argument instead of booleans.
- add AMAP_COPY_NOMERGE flag, and use it for uvm_map_extract.
  PR/32806 from Julio M. Merino Vidal.

remove the following options.  no objections on tech-kern@.

	UVM_PAGER_INLINE
	UVM_AMAP_INLINE
	UVM_PAGE_INLINE
	UVM_MAP_INLINE

sync with head.

Pull up following revision(s) (requested by chs in ticket #10230):
	sys/uvm/uvm_amap.c: revision 1.66 via patch
in amap_cow_now(), handle the case where we have to sleep and some of the
already-copied pages are paged out.  anons that have already been copied
will have refcount == 1, whereas anons that still need to be copied will
have refcount > 1.  fixes PR 25392, PR 30257, PR 31924.

Pull up following revision(s) (requested by chs in ticket #10230):
	sys/uvm/uvm_amap.c: revision 1.66 via patch
in amap_cow_now(), handle the case where we have to sleep and some of the
already-copied pages are paged out.  anons that have already been copied
will have refcount == 1, whereas anons that still need to be copied will
have refcount > 1.  fixes PR 25392, PR 30257, PR 31924.

Pull up following revision(s) (requested by chs in ticket #10230):
	sys/uvm/uvm_amap.c: revision 1.66 via patch
in amap_cow_now(), handle the case where we have to sleep and some of the
already-copied pages are paged out.  anons that have already been copied
will have refcount == 1, whereas anons that still need to be copied will
have refcount > 1.  fixes PR 25392, PR 30257, PR 31924.

- uvm_fault: move a common code of 1B and 2B to a new function.
  don't attempt to allocate anons with kernel_map locked.  PR/32543.
- amap_copy: add an assertion.

in amap_alloc(), only put the amap on the list of amaps if we succeeded
in allocating it.

Remove leading __ from __(const|inline|signed|volatile) -- it is obsolete.

merge ktrace-lwp.

Sync with HEAD. Here we go again...

Pullup via patch (requested in ticket #939 by chs)

  In amap_cow_now(), handle the case where we have to sleep and some of the
  already-copied pages are paged out.  anons that have already been copied
  will have refcount == 1, whereas anons that still need to be copied will
  have refcount > 1. PR#25392, PR#30257, PR#31924

in amap_cow_now(), handle the case where we have to sleep and some of the
already-copied pages are paged out.  anons that have already been copied
will have refcount == 1, whereas anons that still need to be copied will
have refcount > 1.  fixes PR 25392, PR 30257, PR 31924.

wrap swap related code by #ifdef VMSWAP.  always #define VMSWAP for now.

revert "defflag VMSWAP" changes for now.
there seems to be far more people who don't want to edit
their kernel config files than i thought.

defflag VMSWAP.

Use ANSI function decls.

(try to) merge map entries in fault handler.

allocate anons on-demand, rather than reserving static amount of
them on boot/swapon.

- amap_extend: don't extend amap beyond UVM_AMAP_LARGE.
- uvm_map_enter: if we fail to extend amap, just give up merging instead of
  bailing out immediately.

sync with -current

amap_wipeout: remove a comment which is no longer true.
despite of what comment said, i left preempt() call
because i don't think of any bad effects.

Pull up revision 1.57 (requested by chs in ticket #1137):
hack around a UVM problem that causes hangs when large processes fork.
see PR 26908 for details.

Pull up revision 1.57 (requested by chs in ticket #1137):
hack around a UVM problem that causes hangs when large processes fork.
see PR 26908 for details.

sync with head.

Sync with HEAD.

hack around a UVM problem that causes hangs when large processes fork.
see PR 26908 for details.

Sync with HEAD.

for in-kernel maps,
- allocate kva for vm_map_entry from the map itsself and
  remove the static limit, MAX_KMAPENT.
- keep merged entries for later splitting to fix allocate-to-free problem.
  PR/24039.

Fix the sync with head I botched.

Sync with HEAD.

Sync with HEAD

add assertions.

Initialise (most) pools from a link set instead of explicit calls
to pool_init.  Untouched pools are ones that either in arch-specific
code, or aren't initialiased during initial system startup.

 Convert struct session, ucred and lockf to pools.

- Nuke __P().
- Drop trailing spaces.

Pull up revision 1.45 (requested by skrll):
add a new km flag UVM_KMF_CANFAIL, which causes uvm_km_kmemalloc() to
return failure if swap is full and there are no free physical pages.
have malloc() use this flag if M_CANFAIL is passed to it.
use M_CANFAIL to allow amap_extend() to fail when memory is scarce.
this should prevent most of the remaining hangs in low-memory situations.

Add extensible malloc types, adapted from FreeBSD.  This turns
malloc types into a structure, a pointer to which is passed around,
instead of an int constant.  Allow the limit to be adjusted when the
malloc type is defined, or with a function call, as suggested by
Jonathan Stone.

amap_copy: remove stray amap_unlock().

Merge the nathanw_sa branch.

In the SA universe, the switch-to-this-LWP decision is made at a
different level than where preempt() calls are made, which renders
the "newlwp" argument useless.  Replace it with a "more work to do"
boolean argument.  Returning to userspace preempt() calls pass 0.
"Voluntary" preemptions in e.g. uiomove() pass 1.  This will be used
to indicate to the SA subsystem that the LWP is not yet finished in
the kernel.

Collapse the SA vs. non-SA cases of preempt() together, making the
conditional code block much smaller, and don't call sa_preempt() if
more work is to come.

NOTE: THIS IS NOT A COMPLETE FIX TO THE preempt()-in-uiomove() PROBLEM
THAT CURRENTLY EXISTS FOR SA PROCESSES.

Sync with HEAD.

Properly set page references counts at the start of the newly
allocated ppref data to zero in the case of an amap that has empty
space at the front.

Don't set anything in the ppref array if "len" is zero.

Many thanks to Sami Kantoluoto for providing gdb access to a machine
that would reliably crash with problems related to the above, and to
Stephan Thesing for corroborating that the patch properly addressed
the problem.

Note that the ar_pageoff (and related variables) types must be changed
soon.  The use of "int" here is not theoretically sufficient.

Sync with HEAD.

Change uvm_km_kmemalloc() to accept flag UVM_KMF_NOWAIT and pass it to
uvm_map(). Change uvm_map() to honnor UVM_KMF_NOWAIT. For this, change
amap_extend() to take a flags parameter instead of just boolean for
direction, and introduce AMAP_EXTEND_FORWARDS and AMAP_EXTEND_NOWAIT flags
(AMAP_EXTEND_BACKWARDS is still defined as 0x0, to keep the code easier to
read).
Add a flag parameter to uvm_mapent_alloc().
This solves a problem a pool_get(PR_NOWAIT) could trigger a pool_get(PR_WAITOK)
in uvm_mapent_alloc().
Thanks to Chuck Silvers, enami tsugutomo, Andrew Brown and Jason R Thorpe
for feedback.

Properly free "newppref", instead of "amap->am_ppref" (oops), and
delay freeing the old am_ppref so that if we bail early due to
malloc() failures, valid ppref data hasn't been freed for no reason.

Based on comments from enami.

Implement backwards extension of amaps.  There are three cases to deal
with:

Case #1 -- adjust offset: The slot offset in the aref can be
decremented to cover the required size addition.

Case #2 -- move pages and adjust offset: The slot offset is not large
enough, but the amap contains enough inactive space *after* the mapped
pages to make up the difference, so active slots are slid to the "end"
of the amap, and the slot offset is, again, adjusted to cover the
required size addition.  This optimizes for hitting case #1 again on
the next small extension.

Case #3 -- reallocate, move pages, and adjust offset: There is not
enough inactive space in the amap, so the arrays are reallocated, and
the active pages are copied again to the "end" of the amap, and the
slot offset is adjusted to cover the required size.  This also
optimizes for hitting case #1 on the next backwards extension.

This provides the missing piece in the "forward extension of
vm_map_entries" logic, so the merge failure counters have been
removed.

Not many applications will make any use of this at this time (except
for jvms and perhaps gcc3), but a "top-down" memory allocator will use
it extensively.

sync kqueue with -current; this includes merge of gehenna-devsw branch,
merge of i386 MP branch, and part of autoconf rototil work

Catch up to -current.

add a new km flag UVM_KMF_CANFAIL, which causes uvm_km_kmemalloc() to
return failure if swap is full and there are no free physical pages.
have malloc() use this flag if M_CANFAIL is passed to it.
use M_CANFAIL to allow amap_extend() to fail when memory is scarce.
this should prevent most of the remaining hangs in low-memory situations.

sync kqueue branch with HEAD

Catch up to -current.

catch up with -current.

rearrange a few lines to appease an assertion.

Curproc->curlwp renaming.

Change uses of "curproc->l_proc" back to "curproc", which is more like the
original use. Bare uses of "curproc" are now "curlwp".

"curproc" is now #defined in proc.h as ((curlwp) ? (curlwp)->l_proc) : NULL)
so that it is always safe to reference curproc (*de*referencing curproc
is another story, but that's always been true).

catch up with -current on kqueue branch

Catch up to -current.
(CVS: It's not just a program. It's an adventure!)

In amap_pp_adjref(), avoid incorrectly merging the first two chunks in
a ppref array when the range being adjusted includes the beginning of
the array.

Catch up with -current.

Make the amap lock an adaptive mutex.

Pool deals fairly well with physical memory shortage, but it doesn't
deal with shortages of the VM maps where the backing pages are mapped
(usually kmem_map).  Try to deal with this:

* Group all information about the backend allocator for a pool in a
  separate structure.  The pool references this structure, rather than
  the individual fields.
* Change the pool_init() API accordingly, and adjust all callers.
* Link all pools using the same backend allocator on a list.
* The backend allocator is responsible for waiting for physical memory
  to become available, but will still fail if it cannot callocate KVA
  space for the pages.  If this happens, carefully drain all pools using
  the same backend allocator, so that some KVA space can be freed.
* Change pool_reclaim() to indicate if it actually succeeded in freeing
  some pages, and use that information to make draining easier and more
  efficient.
* Get rid of PR_URGENT.  There was only one use of it, and it could be
  dealt with by the caller.

From art@openbsd.org.

Catch up to -current.

in amap_pp_adjref(), avoid unnecessary fragmentation of the am_ppref array
by merging the first changed chunk with the last unchanged chunk if possible.

Sync kqueue branch with -current.

Catch up to -current.

When initially allocating or extending arrays in struct uvm_amap,
adjust allocation size using malloc_roundup().  This eliminates many
unnecessary malloc/memcpy calls.

No need to zero clear after amap->am_bckptr[amap->am_nslot], since we're
clearing corresponding elements in an array amap->am_anon[].

fix bug in amap_wiperange() detected by enami tsugutomo.
loop control was wrong in one case.

Catch up to -current.

Sync the thorpej-mips-cache branch with -current.

add RCSIDs, and in some cases, slightly cleanup #include order

Change some unsigned int variables and parameters to plain ints so
that all usages of those agree on unsigned vs. signed.

Catch up with -current.

curproc->p_cpu ==> curproc->l_cpu

Catch up to -current.

work around swap-space/extent performance problem which causes
long pauses when processes with lots of swapped-out pages exit.

a whole bunch of changes to improve performance and robustness under load:

 - remove special treatment of pager_map mappings in pmaps.  this is
   required now, since I've removed the globals that expose the address range.
   pager_map now uses pmap_kenter_pa() instead of pmap_enter(), so there's
   no longer any need to special-case it.
 - eliminate struct uvm_vnode by moving its fields into struct vnode.
 - rewrite the pageout path.  the pager is now responsible for handling the
   high-level requests instead of only getting control after a bunch of work
   has already been done on its behalf.  this will allow us to UBCify LFS,
   which needs tighter control over its pages than other filesystems do.
   writing a page to disk no longer requires making it read-only, which
   allows us to write wired pages without causing all kinds of havoc.
 - use a new PG_PAGEOUT flag to indicate that a page should be freed
   on behalf of the pagedaemon when it's unlocked.  this flag is very similar
   to PG_RELEASED, but unlike PG_RELEASED, PG_PAGEOUT can be cleared if the
   pageout fails due to eg. an indirect-block buffer being locked.
   this allows us to remove the "version" field from struct vm_page,
   and together with shrinking "loan_count" from 32 bits to 16,
   struct vm_page is now 4 bytes smaller.
 - no longer use PG_RELEASED for swap-backed pages.  if the page is busy
   because it's being paged out, we can't release the swap slot to be
   reallocated until that write is complete, but unlike with vnodes we
   don't keep a count of in-progress writes so there's no good way to
   know when the write is done.  instead, when we need to free a busy
   swap-backed page, just sleep until we can get it busy ourselves.
 - implement a fast-path for extending writes which allows us to avoid
   zeroing new pages.  this substantially reduces cpu usage.
 - encapsulate the data used by the genfs code in a struct genfs_node,
   which must be the first element of the filesystem-specific vnode data
   for filesystems which use genfs_{get,put}pages().
 - eliminate many of the UVM pagerops, since they aren't needed anymore
   now that the pager "put" operation is a higher-level operation.
 - enhance the genfs code to allow NFS to use the genfs_{get,put}pages
   instead of a modified copy.
 - clean up struct vnode by removing all the fields that used to be used by
   the vfs_cluster.c code (which we don't use anymore with UBC).
 - remove kmem_object and mb_object since they were useless.
   instead of allocating pages to these objects, we now just allocate
   pages with no object.  such pages are mapped in the kernel until they
   are freed, so we can use the mapping to find the page to free it.
   this allows us to remove splvm() protection in several places.

The sum of all these changes improves write throughput on my
decstation 5000/200 to within 1% of the rate of NetBSD 1.5
and reduces the elapsed time for "make release" of a NetBSD 1.5
source tree on my 128MB pc to 10% less than a 1.5 kernel took.

Catch up with -current.

update to -current

seperate -> separate

Catch up to -current.

replace vm_map{,_entry}_t with struct vm_map{,_entry} *.

remove trailing whitespace.

Sync with HEAD.

clean up DIAGNOSTIC checks, use KASSERT().

Sync with HEAD.

Change uvm_analloc() to return a locked anon, update all callers,
and fix an anon locking protocol error in uvm_loanzero().

Sprinkle some assertions:
amap_free(): Assert that the amap is locked.
amap_share_protect(): Assert that the amap is locked.
amap_wipeout(): Assert that the amap is locked.
uvm_anfree(): Assert that the anon has a reference count of 0 and is
              not locked.
uvm_anon_lockloanpg(): Assert that the anon is locked.
anon_pagein(): Assert that the anon is locked.
uvmfault_anonget(): Assert that the anon is locked.
uvm_pagealloc_strat(): Assert that the uobj or the anon is locked

And fix the problems these have uncovered:
amap_cow_now(): Lock the new anon after allocating it, and unref and
                unlock it (rather than lock!) before freeing it in case
                of an error condition.  This should fix a problem reported
		by Dan Carosone using cdrecord on an i386 MP kernel.
uvm_fault(): Case1B -- Lock the new anon afer allocating it, and unlock
             it later when we unlock the old anon.
	     Case2 -- Lock the new anon after allocating it, and unlock
	     it later by passing it to uvmfault_unlockall() (we set anon
	     to NULL if we're not doing a promote fault).

Sync with HEAD.

lots of cleanup:
use queue.h macros and KASSERT().
address amap offsets in pages instead of bytes.
make amap_ref() and amap_unref() take an amap, offset and length
  instead of a vm_map_entry_t.
improve whitespace and comments.

Update thorpej_scsipi to -current as of a month ago

MALLOC()/FREE() are not to be used for variable size allocations.

Pull up rev. 1.25 (approved by jhawk):
Fix a fairly obvious locking error in amap_cow_now() -- the amap was
left locked upon exit from the function (how did this one slip for
so long?)

Fix a fairly obvious locking error in amap_cow_now() -- the amap was
left locked upon exit from the function (how did this one slip for
so long?)

remove include of <vm/vm.h>

remove/move more mach vm header files:

	<vm/pglist.h> -> <uvm/uvm_pglist.h>
	<vm/vm_inherit.h> -> <uvm/uvm_inherit.h>
	<vm/vm_kern.h> -> into <uvm/uvm_extern.h>
	<vm/vm_object.h> -> nothing
	<vm/vm_pager.h> -> into <uvm/uvm_pager.h>

also includes a bunch of <vm/vm_page.h> include removals (due to redudancy
with <vm/vm.h>), and a scattering of other similar headers.

bring in latest KAME (as of 19991130, KAME/NetBSD141) into kame branch
just for reference purposes.
This commit includes 1.4 -> 1.4.1 sync for kame branch.

The branch does not compile at all (due to the lack of ALTQ and some other
source code).  Please do not try to modify the branch, this is just for
referenre purposes.

synchronization to latest KAME will take place on HEAD branch soon.

eliminate the PMAP_NEW option by making it required for all ports.
ports which previously had no support for PMAP_NEW now implement
the pmap_k* interfaces as wrappers around the non-k versions.

Update from trunk.

pullup 1.20->1.21 (cgd)

fix allocation handling bugs in amap_alloc1().  if the first or second
sub-structure malloc() failed, it was quite likely that the function
would return success incorrectly.  This is this direct cause of the bug
reported in PR#7897.  (Thanks to chs for helping to track it down.)

merge everything from chs-ubc branch.

pull up 1.19 -> 1.20:
add a `flags' argument to uvm_pagealloc_strat().
define a flag UVM_PGA_USERESERVE to allow non-kernel object
allocations to use pages from the reserve.
use the new flag for allocations in pmap modules.

add a `flags' argument to uvm_pagealloc_strat().
define a flag UVM_PGA_USERESERVE to allow non-kernel object
allocations to use pages from the reserve.
use the new flag for allocations in pmap modules.

thread_wakeup() -> wakeup().

comment cleanup, shift around the inline stuff a bit,
rename VM_AMAP_PPREF (to UVM_AMAP_PPREF).

cleanup/reorg:
- break anon related functions out of uvm_amap.c and put them in their own
  file (uvm_anon.c).  includes break up uvm_anon_init into an amap and an
  an anon init function
- ensure that only functions within the amap module access amap structure
  fields (add macros to amap api as needed)

initial snapshot.  lots left to do.

be consistent with locking of amaps and anons when freeing them.

shift by PAGE_SHIFT instead of multiplying or dividing by PAGE_SIZE.

fix ppref botch.  establish ppref at split time before we add the duplicate
reference.

Allocate vm_anon arrays from kernel_map, not via MALLOC().  Helps relieve
much of UVM's kmem_map usage.

Use the pool allocator (and the "nointr" pool page allocator) for
vm_amap structures.

Merge paddr_t changes into the main branch.

bzero->memset, bcopy->memcpy, bcmp->memcmp

Split vm_offset_t and vm_size_t into paddr_t, psize_t, vaddr_t, and vsize_t.

add a "<-done!" log

detect ending VA wrap-around in the chunking code of amap_copy.

fixes problem reported by Ken Nakata <kenn@synap.ne.jp> on the mac68k
where the stack amap chunking caused entry->end to wrap around to zero,
thus corrupting the map entry list and causing kmem_map to fill.

Remove inclusions of syscall (and syscall argument) related header files;
we don't need them here.

KNF.

- add defopt's for UVM, UVMHIST and PMAP_NEW.
- remove unnecessary UVMHIST_DECL's.

KNF

restore rcsids

fix typoes in locking.
use M_UVMAMAP instead of M_TEMP for malloc type.

RCS ID police.

initial import of the new virtual memory system, UVM, into -current.

UVM was written by chuck cranor <chuck@maria.wustl.edu>, with some
minor portions derived from the old Mach code.  i provided some help
getting swap and paging working, and other bug fixes/ideas.  chuck
silvers <chuq@chuq.com> also provided some other fixes.

this is the UVM kernel code portion.


this will be KNF'd shortly.  :-)

Initial revision