CVS log for src/sys/uvm/uvm_init.c

Up to [cvs.NetBSD.org] / src / sys / uvm

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

Repply this change with a couple of bugs fixed:

- Do away with separate pool_cache for some kernel objects that have no special
  requirements and use the general purpose allocator instead. On one of my
  test systems this makes for a small (~1%) but repeatable reduction in system
  time during builds presumably because it decreases the kernel's cache /
  memory bandwidth footprint a little.
- vfs_lockf: cache a pointer to the uidinfo and put mutex in the data segment.

Back out recent change to replace pool_cache with then general allocator.
Will return to this when I have time again.

- Do away with separate pool_cache for some kernel objects that have no special
  requirements and use the general purpose allocator instead.  On one of my
  test systems this makes for a small (~1%) but repeatable reduction in system
  time during builds presumably because it decreases the kernel's cache /
  memory bandwidth footprint a little.
- vfs_lockf: cache a pointer to the uidinfo and put mutex in the data segment.

uvm(9): One rndsource for faults -- not one per CPU.

All relevant state is per-CPU anyway; the only substantive difference
this makes is how many entries appear in `rndctl -l' output and what
they are called -- formerly the somewhat confusing `cpuN', meaning
`page faults on cpuN', and now just `uvmfault'.  I don't think
there's any real value in being able to enable or disable measurement
or counting of page faults on one CPU vs others, so although this
could be a minor compatibility change, it's hard to imagine it
matters much.

XXX kernel ABI change in struct cpu_info

Sync w/ HEAD.

In uvmpd_tryownerlock(), if the initial try-lock of the owner lock fails
then rather than do more try-locks and eventually sleep for a tick,
take a hold on the current owner's lock, drop the page interlock,
and acquire the lock that we took the hold on in a blocking fashion.
After we get the lock, check if the lock that we acquired is still
the lock for the owner of the page that we're interested in.
If the owner hasn't changed then can proceed with this page,
otherwise we will skip this page and move on to a different page.
This dramatically reduces the amount of time that the pagedaemon
sleeps trying to get locks, since even 1 tick is an eternity to sleep
in this context and it was easy to trigger that case in practice,
and with this new method the pagedaemon only very rarely actually blocks
to acquire the lock that it wants since the object locks are adaptive,
and when the pagedaemon does block then the amount of time it spends
sleeping will be generally be much less than 1 tick.

Add a new, more aggressive allocator for uvm_pglistalloc() to allocate
contiguous physical pages, and try this new allocator if the existing
one fails.  The existing contig allocator only tries to allocate pages
that are already free, which works fine shortly after boot but rarely
works after the system has been up for a while.  The new allocator uses
the pagedaemon to evict pages from memory in the hope that this will
free up a range of pages that satisfies the constraits of the request.
This should help with things like plugging in a USB device, which often
fails for some USB controllers because they can't get contigous memory.

Merge changes from current as of 20200406

Fix a comment.

Redo the page allocator to perform better, especially on multi-core and
multi-socket systems.  Proposed on tech-kern.  While here:

- add rudimentary NUMA support - needs more work.
- remove now unused "listq" from vm_page.

Break the global uvm_pageqlock into a per-page identity lock and a private
lock for use of the pagedaemon policy code.  Discussed on tech-kern.

PR kern/54209: NetBSD 8 large memory performance extremely low
PR kern/54210: NetBSD-8 processes presumably not exiting
PR kern/54727: writing a large file causes unreasonable system behaviour

Give each of the page queue locks their own cache line.

Sync with HEAD

Remove emap support. Unfortunately it never got to state where it would be
used and usable, due to reliability and limited & complicated MD support.

Going forward, we need to concentrate on interface which do not map anything
into kernel in first place (such as direct map or KVA-less I/O), rather
than making those mappings cheaper to do.

update from HEAD

Sync with HEAD

Sync with HEAD.  (Note that most of these changes are simply $NetBSD$
tag issues.)

"Make NetBSD great again!"

Introduce uvm_hotplug(9) to the kernel.

Many thanks, in no particular order to:

TNF, for funding the project.

Chuck Silvers - for multiple API reviews and feedback.
Nick Hudson - for testing on multiple architectures and bugfix patches.
Everyone who helped with boot testing.

KeK (http://www.kek.org.in) for hosting the primary developers.

Add a new function called uvm_md_init() that can be called at the
appropriate time in the boot path by MD code.

Pull up following revision(s) (requested by mrg in ticket #718):
	sys/arch/x86/include/pmap.h: revision 1.56
	sys/arch/x86/x86/pmap.c: revision 1.188
	sys/dev/pci/agp_amd64.c: revision 1.8
	sys/dev/pci/agp_i810.c: revision 1.118
	sys/external/bsd/drm2/dist/drm/i915/i915_dma.c: revision 1.16
	sys/external/bsd/drm2/dist/drm/i915/i915_gem.c: revision 1.29
	sys/external/bsd/drm2/dist/drm/nouveau/nouveau_agp.c: revision 1.3
	sys/external/bsd/drm2/dist/drm/nouveau/nouveau_ttm.c: revision 1.4
	sys/external/bsd/drm2/dist/drm/radeon/atombios_crtc.c: revision 1.3
	sys/external/bsd/drm2/dist/drm/radeon/radeon_agp.c: revision 1.3
	sys/external/bsd/drm2/dist/drm/radeon/radeon_display.c: revision 1.3
	sys/external/bsd/drm2/dist/drm/radeon/radeon_legacy_crtc.c: revision 1.2
	sys/external/bsd/drm2/dist/drm/radeon/radeon_object.c: revision 1.3
	sys/external/bsd/drm2/dist/drm/radeon/radeon_ttm.c: revision 1.7
	sys/external/bsd/drm2/dist/drm/ttm/ttm_bo.c: revisions 1.7-1.10
	sys/external/bsd/drm2/dist/drm/ttm/ttm_bo_util.c: revision 1.5
	sys/external/bsd/drm2/i915drm/intelfb.c: revision 1.13
	sys/external/bsd/drm2/include/drm/drm_wait_netbsd.h: revisions 1.12, 1.13
	sys/external/bsd/drm2/include/linux/mm.h: revision 1.5
	sys/external/bsd/drm2/include/linux/pci.h: revisions 1.16, 1.17
	sys/external/bsd/drm2/nouveau/nouveaufb.c: revision 1.2
	sys/external/bsd/drm2/radeon/radeon_pci.c: revisions 1.8, 1.9
	sys/uvm/uvm_init.c: revision 1.46
Hack against the blank console problem:
Leave the CLUT alone on ancient cards. At least this leaves us with a
semi working console (red and blue are flipped). Leave an example of what
seems to be happening but disable it because colors are better than 444 bit
greyscale.
--
Initialize P->V tracking for unmanaged device pages in uvm_init.

Conditional on __HAVE_PMAP_PV_TRACK until we add it to all pmaps.

MI part of pmap_pv(9) change proposed on tech-kern:

https://mail-index.netbsd.org/tech-kern/2015/03/26/msg018561.html
--
Implement pmap_pv(9) for x86 for P->V tracking of unmanaged pages.

Proposed on tech-kern with no objections:

https://mail-index.netbsd.org/tech-kern/2015/03/26/msg018561.html
--
Use pmap_pv(9) to remove mappings of Intel graphics aperture pages.

Proposed on tech-kern with no objections:

https://mail-index.netbsd.org/tech-kern/2015/03/26/msg018561.html

Further background at:

https://mail-index.netbsd.org/tech-kern/2014/07/23/msg017392.html
--
Use pmap_pv(9) to remove mappings of device pages in TTM.

Adapt nouveau and radeon to do pmap_pv_track for their device pages.

Proposed on tech-kern with no objections:

https://mail-index.netbsd.org/tech-kern/2015/03/26/msg018561.html

Further background at:

https://mail-index.netbsd.org/tech-kern/2014/07/23/msg017392.html
--
Fix error branches in agp_amd64.c.

- agp_generic_detach always.
- Free asc if it was allocated.  (Found by Brainy, noted by maxv@.)
- Free the GATT if it was allocated.
--
pmf_device_register returns false on failure, not true
--
In DRM_SPIN_WAIT_ON, don't stop after waiting only one tick.

Continue the loop to recheck the condition and count the whole
duration.
--
Don't use the video BIOS memory as an i915 flush page!
--
Don't let anyone else allocate the video BIOS either.
--
Missed a zero: it's 0x100000, not 0x10000.
--
Don't reserve if atomic -- caller must have pre-pinned the buffer.
--
Don't reserve if atomic -- caller must have pre-pinned the buffer.
--
almost add radeondrmkms suspend/resume support.  it unfortunately doesn't work.
--
Need the page's uvm object lock to do pmap_page_protect.
--
Use KASSERTMSG to show bad base/offset.
--
KASSERT about page-alignment on initialization too.
--
Don't break when hardclock_ticks wraps around.

Since we now only count time spent in wait, rather than determining
the end time and checking whether we've passed it, timeouts might be
marginally longer in effect.  Unlikely to be an issue.
--
Remove broken drm2 vm_mmap stub.  Can't possibly have ever worked.
--
apply some of the additional changes from Arto Huusko in PR#49645:
- call pmf_device_deregister on detach.

i've kept the "resume = true" for radeon_resume_kms() call as it
seems to work for me (indeed, code inspection shows it is unused
on netbsd :-)

my old nforce4 box that can resume old drm (or could, last i tried
several years ago) while X and GL apps were running, can at least
survive a resume if X hasn't started.  my one attempt so far with
X exited, but having run, did not work.
--
First attempt to make ttm_buffer_object_transfer less bogus.
--
Make sure mem.bus.is_iomem is initialized.  PR 49833

Sync with HEAD

Initialize P->V tracking for unmanaged device pages in uvm_init.

Conditional on __HAVE_PMAP_PV_TRACK until we add it to all pmaps.

MI part of pmap_pv(9) change proposed on tech-kern:

https://mail-index.netbsd.org/tech-kern/2015/03/26/msg018561.html

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs.  ("Protocol error: too many arguments")

resync with head

improve on comments

sync with head

Move the uvm_scheduler_mutex and cv init to uvm_init since they are
independent of VMSWAP.

merge to -current.

Make sure to export uvmexp_* if MODULAR is defined.
Make the uvmexp_page* be a pointer to a const int as well as having the
pointer be const as well.

Major changes to uvm.
Support multiple collections (groups) of free pages and run the page
reclaimation algorithm on each group independently.

pool_page_alloc, pool_page_alloc_meta: avoid extra compare, use const.
ffs_mountfs,sys_swapctl: replace memset with kmem_zalloc.
sys_swapctl: move kmem_free outside the lock path.
uvm_init: fix comment, remove pointless numeration of steps.
uvm_map_enter: remove meflagval variable.
Fix some indentation.

extending vmem(9) to be able to allocated resources for it's own needs.
simplifying uvm_map handling (no special kernel entries anymore no relocking)
make malloc(9) a thin wrapper around kmem(9)
(with private interface for interrupt safety reasons)

releng@ acknowledged

- fix page loaning  XXX make O->A loaning further
- add some statistics

Do not call uvm_loan_init() twice.

Sync with HEAD.

sync with head

Initialize UVM loaning subsystem a bit later, after kmem(9).
Makes UVMHIST work again.

Replace "malloc" in comments, remove unnecessary header inclusions.

sync with head

Sync with HEAD

udpate license clauses on my code to match the new-style BSD licenses.
based on diff that rmind@ sent me.

no functional change with this commit.

Sync with HEAD.

Oops.  Fix thinko.

Platforms that dynamically set PAGE_{SIZE,MASK,SHIFT}, those values are
saved in struct uvmexp.  Expose only the relevant part for symbol users,
so that they don't need to include the whole uvm(9) API.

Change struct uvm_object::vmobjlock to be dynamically allocated with
mutex_obj_alloc().  It allows us to share the locks among UVM objects.

sync with head

Remove uarea swap-out functionality:

- Addresses the issue described in PR/38828.
- Some simplification in threading and sleepq subsystems.
- Eliminates pmap_collect() and, as a side note, allows pmap optimisations.
- Eliminates XS_CTL_DATA_ONSTACK in scsipi code.
- Avoids few scans on LWP list and thus potentially long holds of proc_lock.
- Cuts ~1.5k lines of code.  Reduces amd64 kernel size by ~4k.
- Removes __SWAP_BROKEN cases.

Tested on x86, mips, acorn32 (thanks <mpumford>) and partly tested on
acorn26 (thanks to <bjh21>).

Discussed on <tech-kern>, reviewed by <ad>.

Sync with HEAD.

sync with head.

Ephemeral mapping (emap) implementation.  Concept is based on the idea that
activity of other threads will perform the TLB flush for the processes using
emap as a side effect.  To track that, global and per-CPU generation numbers
are used.  This idea was suggested by Andrew Doran; various improvements to
it by me.  Notes:

- For now, zero-copy on pipe is not yet enabled.
- TCP socket code would likely need more work.
- Additional UVM loaning improvements are needed.

Proposed on <tech-kern>, silence there.
Quickly reviewed by <ad>.

sync with head.

Sync with HEAD.

Sync with HEAD.

- Initialize pool subsystem and kmem(9) earlier, when UVM is up enough.
- Remove uao_hashinit() workaround used for anon-objects.
- Replace malloc with kmem.

OK by <yamt>.

Sync w/ -current. 34 merge conflicts to follow.

sync with head.

Sync with HEAD.

Also fix build.

Replace the global vm_page hash with a per vm_object rbtree.
Proposed on tech-kern@.

sync with HEAD

Sync with HEAD.

sync with head.

remove a special allocator for uareas, which is no longer necessary.
use pool_cache instead.

sync with head

sync with HEAD

Sync with HEAD

Merge vmlocking2 to head.

Lock readahead context using the associated object's lock.

Pull the vmlocking changes into a new branch.

Sync with HEAD.

Sync with HEAD

sync with head.

Sync with HEAD.

call debug_init earlier.  ie. before malloc is used.

Sync with HEAD.

sync with head.

Sync with HEAD.

fix some races around pagedaemon and uvm_wait.  ok'ed by Andrew Doran.

Sync with HEAD.

Make the uarea cache per-CPU and drain in batches of 4.

Sync with HEAD.

Merge unobtrusive locking changes from the vmlocking branch.

file uvm_init.c was added on branch matt-mips64 on 2007-07-21 19:21:55 +0000

Merge unobtrusive locking changes from the vmlocking branch.

Sync with head.

Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

Split uvm_hashlock into an array of 32 locks.

- Put a per-LWP lock around swapin / swapout.
- Replace use of lockmgr().
- Minor locking fixes and assertions.
- uvm_map.h no longer pulls in proc.h, etc.
- Use kpause where appropriate.

Pull in the initial set of changes for the vmlocking branch.

sync with head.

Sync with head.

merge yamt-pdpolicy branch.
	- separate page replacement policy from the rest of kernel
	- implement an alternative replacement policy

make UVM_KICK_PDAEMON() a real function and stop including
uvm_pdpolicy.h from uvm.h.  this also fixes build of pmap(1).

sync with head

sync with head.

sync with head.

Sync with head.

Sync with head.

move wait points for kva from upper layers to vm_map.  PR/33185 #1.

XXX there is a concern about interaction with kva fragmentation.
see: http://mail-index.NetBSD.org/tech-kern/2006/05/11/0000.html

separate page replacement policy from the rest of kernel.

merge ktrace-lwp.

Sync with HEAD. Here we go again...

Use ANSI function decls.

allocate anons on-demand, rather than reserving static amount of
them on boot/swapon.

sync with -current

Sync with HEAD.

move the call to link_pool_init() to the end of uvm_init().  needed for sun3.

Fix the sync with head I botched.

Sync with HEAD.

Sync with HEAD

Initialise (most) pools from a link set instead of explicit calls
to pool_init.  Untouched pools are ones that either in arch-specific
code, or aren't initialiased during initial system startup.

 Convert struct session, ucred and lockf to pools.

update comment - kmem_map is created in kmeminit(), not uvm_km_init()

Back out the following chagne:
    http://mail-index.netbsd.org/source-changes/2003/05/08/0068.html

There were some side-effects that I didn't anticipate, and fixing them
is proving to be more difficult than I thought, do just eject for now.
Maybe one day we can look at this again.

Fixes PR kern/21517.

Simplify the way the bounds of the managed kernel virtual address
space is advertised to UVM by making virtual_avail and virtual_end
first-class exported variables by UVM.  Machine-dependent code is
responsible for initializing them before main() is called.  Anything
that steals KVA must adjust these variables accordingly.

This reduces the number of instances of this info from 3 to 1, and
simplifies the pmap(9) interface by removing the pmap_virtual_space()
function call, and removing two arguments from pmap_steal_memory().

This also eliminates some kludges such as having to burn kernel_map
entries on space used by the kernel and stolen KVA.

This also eliminates use of VM_{MIN,MAX}_KERNEL_ADDRESS from MI code,
this giving MD code greater flexibility over the bounds of the managed
kernel virtual address space if a given port's specific platforms can
vary in this regard (this is especially true of the evb* ports).

Fix the following pathological scanario:
* User allocates ZFOD region, but does not actually touch the buffer
  to fault in the pages.
* In a loop, user writes this buffer to a network socket, triggering
  sosend_loan().
* uvm_loan() calls uvm_loanzero() once for each page in the loaned
  region (since the pages have not yet faulted in).  This causes a
  page to be allocated and zero'd.  The result is the kernel spends
  a lot of time allocating and zero'ing pages.

This fixes creates a special object which owns a single zero'd page.
This single zero'd page is used to satisfy all loans of non-resident
ZFOD mappings.

Thanks to Allen Briggs for discovering the problem and for providing
an initial patch.

Catch up to -current.

Sync kqueue branch with -current.

Catch up to -current.

Sync the thorpej-mips-cache branch with -current.

add RCSIDs, and in some cases, slightly cleanup #include order

Initial commit of scheduler activations and lightweight process support.

Update thorpej_scsipi to -current as of a month ago

remove include of <vm/vm.h>

remove/move more mach vm header files:

	<vm/pglist.h> -> <uvm/uvm_pglist.h>
	<vm/vm_inherit.h> -> <uvm/uvm_inherit.h>
	<vm/vm_kern.h> -> into <uvm/uvm_extern.h>
	<vm/vm_object.h> -> nothing
	<vm/vm_pager.h> -> into <uvm/uvm_pager.h>

also includes a bunch of <vm/vm_page.h> include removals (due to redudancy
with <vm/vm.h>), and a scattering of other similar headers.

Don't need to include <sys/conf.h> here.

remove now >1 year old pre-release message.

cleanup/reorg:
- break anon related functions out of uvm_amap.c and put them in their own
  file (uvm_anon.c).  includes break up uvm_anon_init into an amap and an
  an anon init function
- ensure that only functions within the amap module access amap structure
  fields (add macros to amap api as needed)

Merge paddr_t changes into the main branch.

bzero->memset, bcopy->memcpy, bcmp->memcmp

Split vm_offset_t and vm_size_t into paddr_t, psize_t, vaddr_t, and vsize_t.

Remove inclusions of syscall (and syscall argument) related header files;
we don't need them here.

KNF.

- add defopt's for UVM, UVMHIST and PMAP_NEW.
- remove unnecessary UVMHIST_DECL's.

restore rcsids

enable paging of kernel_object.

RCS ID police.

initial import of the new virtual memory system, UVM, into -current.

UVM was written by chuck cranor <chuck@maria.wustl.edu>, with some
minor portions derived from the old Mach code.  i provided some help
getting swap and paging working, and other bug fixes/ideas.  chuck
silvers <chuq@chuq.com> also provided some other fixes.

this is the UVM kernel code portion.


this will be KNF'd shortly.  :-)

Initial revision