CVS log for src/sys/uvm/uvm_pglist.c

Up to [cvs.NetBSD.org] / src / sys / uvm

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

Pull up following revision(s) (requested by tnn in ticket #554):

	sys/uvm/uvm_physseg.c: revision 1.20
	sys/uvm/uvm_pglist.c: revision 1.91
	sys/uvm/uvm_pglist.c: revision 1.92
	sys/uvm/uvm_physseg.h: revision 1.9

uvm: change type of uvm_physseg.start_hint from u_int to u_long
Avoids assertion failure in uvm_pglistalloc_s_ps() with large paddrs.
PR kern/57683.

fix DEBUG build

fix DEBUG build

uvm: change type of uvm_physseg.start_hint from u_int to u_long

Avoids assertion failure in uvm_pglistalloc_s_ps() with large paddrs.
PR kern/57683.

Update uvm_pglistalloc_[cs]_ps to return EINVAL if [low, high] doesn't
match any memory.

Useful for bus_dmamem_alloc where a tag might not cover any memory.
This will be used in an update to ehci.

"looks good" from chuq@

Slight code re-structure and wrap a long line.  Interestingly this gives
the same binary before and after.

Sync with HEAD.

Sync with HEAD.

in uvm_pglistalloc_contig_aggressive(), avoid looking forward past
the end of the target range of the physseg.
fixes PR 56074.

Trailing whitespace

Add a new, more aggressive allocator for uvm_pglistalloc() to allocate
contiguous physical pages, and try this new allocator if the existing
one fails.  The existing contig allocator only tries to allocate pages
that are already free, which works fine shortly after boot but rarely
works after the system has been up for a while.  The new allocator uses
the pagedaemon to evict pages from memory in the hope that this will
free up a range of pages that satisfies the constraits of the request.
This should help with things like plugging in a USB device, which often
fails for some USB controllers because they can't get contigous memory.

Remove PG_ZERO.  It worked brilliantly on x86 machines from the mid-90s but
having spent an age experimenting with it over the last 6 months on various
machines and with different use cases it's always either break-even or a
slight net loss for me.

Counter tweaks:

- Don't need to count anonpages+filepages any more; clean+unknown+dirty for
  each kind of page can be summed to get the totals.

- Track the number of free pages with a counter so that it's one less thing
  for the allocator to do, which opens up further options there.

- Remove cpu_count_sync_one(). It has no users and doesn't save a whole lot.
  For the cheap option, give cpu_count_sync() a boolean parameter indicating
  that a cached value is okay, and rate limit the updates for cached values
  to hz.

uvm_availmem(): give it a boolean argument to specify whether a recent
cached value will do, or if the very latest total must be fetched.  It can
be called thousands of times a second and fetching the totals impacts not
only the calling LWP but other CPUs doing unrelated activity in the VM
system.

uvm_pglistfree(): just use uvm_pagefree().

Merge changes from current as of 20200406

uvm_pglistalloc() / uvm_pglistfree() musn't be called from interrupt
context.  Assert it.

Sync with head.

Pull up following revision(s) (requested by rin in ticket #732):

	sys/uvm/uvm_pglist.c: revision 1.80

Make this compile again with PGALLOC_VERBOSE.

Make this compile again with PGALLOC_VERBOSE.

Rename uvm_free() -> uvm_availmem().

Redo the page allocator to perform better, especially on multi-core and
multi-socket systems.  Proposed on tech-kern.  While here:

- add rudimentary NUMA support - needs more work.
- remove now unused "listq" from vm_page.

Detangle the pagedaemon from uvm_fpageqlock:

- Have a single lock (uvmpd_lock) to protect pagedaemon state that was
  previously covered by uvmpd_pool_drain_lock plus uvm_fpageqlock.
- Don't require any locks be held when calling uvm_kick_pdaemon().
- Use uvm_free().

- Rename VM_PGCOLOR_BUCKET() to VM_PGCOLOR().  I want to reuse "bucket" for
  something else soon and TBH it matches what this macro does better.

- Add inlines to set/get locator values in the unused lower bits of
  pg->phys_addr.  Begin by using it to cache the freelist index, because
  computing it is expensive and that shows up during profiling.  Discussed
  on tech-kern.

uvmexp.free -> uvm_free()

- Extend the per-CPU counters matt@ did to include all of the hot counters
  in UVM, excluding uvmexp.free, which needs special treatment and will be
  done with a separate commit.  Cuts system time for a build by 20-25% on
  a 48 CPU machine w/DIAGNOSTIC.

- Avoid 64-bit integer divide on every fault (for rnd_add_uint32).

Break the global uvm_pageqlock into a per-page identity lock and a private
lock for use of the pagedaemon policy code.  Discussed on tech-kern.

PR kern/54209: NetBSD 8 large memory performance extremely low
PR kern/54210: NetBSD-8 processes presumably not exiting
PR kern/54727: writing a large file causes unreasonable system behaviour

Sync with HEAD

Sync with HEAD, resolve a couple of conflicts

only warn once per call to uvm_pglistalloc_simple() if waiting.

Sync with HEAD

Resolve a couple of conflicts (result of the uimin/uimax changes)

Rename min/max -> uimin/uimax for better honesty.

These functions are defined on unsigned int.  The generic name
min/max should not silently truncate to 32 bits on 64-bit systems.
This is purely a name change -- no functional change intended.

HOWEVER!  Some subsystems have

	#define min(a, b)	((a) < (b) ? (a) : (b))
	#define max(a, b)	((a) > (b) ? (a) : (b))

even though our standard name for that is MIN/MAX.  Although these
may invite multiple evaluation bugs, these do _not_ cause integer
truncation.

To avoid `fixing' these cases, I first changed the name in libkern,
and then compile-tested every file where min/max occurred in order to
confirm that it failed -- and thus confirm that nothing shadowed
min/max -- before changing it.

I have left a handful of bootloaders that are too annoying to
compile-test, and some dead code:

cobalt ews4800mips hp300 hppa ia64 luna68k vax
acorn32/if_ie.c (not included in any kernels)
macppc/if_gm.c (superseded by gem(4))

It should be easy to fix the fallout once identified -- this way of
doing things fails safe, and the goal here, after all, is to _avoid_
silent integer truncations, not introduce them.

Maybe one day we can reintroduce min/max as type-generic things that
never silently truncate.  But we should avoid doing that for a while,
so that existing code has a chance to be detected by the compiler for
conversion to uimin/uimax without changing the semantics until we can
properly audit it all.  (Who knows, maybe in some cases integer
truncation is actually intended!)

update from HEAD

Sync with HEAD

Sync with HEAD.  (Note that most of these changes are simply $NetBSD$
tag issues.)

PRIxPHYSMEM -> PRIxPHYSSEG to fix the build

Whitespace

"Make NetBSD great again!"

Introduce uvm_hotplug(9) to the kernel.

Many thanks, in no particular order to:

TNF, for funding the project.

Chuck Silvers - for multiple API reviews and feedback.
Nick Hudson - for testing on multiple architectures and bugfix patches.
Everyone who helped with boot testing.

KeK (http://www.kek.org.in) for hosting the primary developers.

Define UVMDEBUG for expensive debugging operations. Idea from chuq.

Don't use C++ try keyword as a variable name.

Rebase to HEAD as of a few days ago.

Rebase.

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs.  ("Protocol error: too many arguments")

Back out previous silliness -- on failure no pages are allocated.

Don't leak memory on failure in uvm_pglistalloc_contig.

Free pages like uvm_pglistalloc_simple does.

Discovered by code inspection.

sync with head

Adapt to K{,D}ASSERTMSG changes

Mark potentialy unused variables

Improve UVM_PAGE_TRKOWN.
Add more asserts to uvm_page.

Add more KASSERTs (more! more! more!).
When returning page to the free pool, make sure to dequeue the pages before
hand or free page queue corruption will happen.

Major changes to uvm.
Support multiple collections (groups) of free pages and run the page
reclaimation algorithm on each group independently.

remove pg->listq and uobj->memq

Modify *ASSERTMSG() so they are now used as variadic macros. The main goal
is to provide routines that do as KASSERT(9) says: append a message
to the panic format string when the assertion triggers, with optional
arguments.

Fix call sites to reflect the new definition.

Discussed on tech-kern@. See
http://mail-index.netbsd.org/tech-kern/2011/09/07/msg011427.html

Sync with HEAD.

Restore $NetBSD$

Rework page free lists to be sorted by color first rather than free_list.
Kept per color PGFL_* counter in each page free list.
Minor cleanups.

sync with head

Fix a bug where limit could be greater avail_end.  Now if that happens, we
just bail.  Use KDASSERTMSG so panics are more informative.

Make uvm_map recognize UVM_FLAG_COLORMATCH which tells uvm_map that the
'align' argument specifies the starting color of the KVA range to be returned.

When calling uvm_km_alloc with UVM_KMF_VAONLY, also specify the starting
color of the kva range returned (UMV_KMF_COLORMATCH) and pass those to
uvm_map.

In uvm_pglistalloc, make sure the pages being returned have sequentially
advancing colors (so they can be mapped in a contiguous address range).
Add a few missing UVM_FLAG_COLORMATCH flags to uvm_pagealloc calls.

Make the socket and pipe loan color-safe.

Make the mips pmap enforce strict page color (color(VA) == color(PA)).

Replace "malloc" in comments, remove unnecessary header inclusions.

sync with head

Sync with HEAD

Introducing inner loop prevent us from exiting from the original loop.

When starting the second pass, don't continue the for loop but instead
just test try exceeding limit.

Use the (new) KDASSERTMSG

Fix start_hint in "simple" alloc (fencepost error).
When restarting the loop, make sure end is not above current limit.
Do a quick test to see if the physseg is within the range of desired addresses.

DEBUG does not imply DIAGNOSTIC; make sure we have a non-null
KASSERTMSG implementation (DIAGNOSTIC) so that the variable inside
the DEBUG section gets used.

Fix the corruption of ps->start_hint.

Cleanup/add some asserts.  no functional change.

buildfix: use PRIxPADDR for type paddr_t

Improve the efficiency of searching for a contiguous set of free pages.

Revert vm_physseg allocation changes.  A report says that it causes
panics when used with mplayer in heavy load.

Sync with HEAD.

build fix: vm_physmem_index is only used with DEBUG.
Fix build when DIAGNOSTIC is enabled but not DEBUG

Optimize DIAGNOSTIC check code.

Fix DIAGNOSTIC physseg find check.

Be a little more friendly to dynamic physical segment registration.

Maintain an array of pointer to struct vm_physseg, instead of struct
array.  So that VM subsystem can take its pointer safely.  Pointer
to this struct will replace raw paddr_t usage in the future.

Dynamic removal is not supported yet.

Only MD data structure changes, no kernel bump needed.

Tested on i386, amd64, powerpc/ibm40x, arm11.

Fix debug code.

Sync with HEAD.

sync with head.

sync with head

disable some DEBUG code uvm_pglist_add() that has severe performance
problems with large mappings.  i've seen my system hang for a total
of 45 seconds when radeondrm is opened by X11, and it is the checks
in this function that take so long.

Fix bad initialization spotted by Manuel Bouyer.

Always use struct vm_physseg *vm_physmem_ptrs[] in MD code.

vm_nphysseg -> vm_nphysmem

Use roundup2 instead of roundup when doing alignment rounding since all
alignments must be a power of 2. (thanks to rmind for suggesting it).

Add a start_hint to vm_physseg so when allocating pages, we can skip
forward over pages that are probably still allocated.

Remove some optimizations since they actually don't do the right thing.
We never want to test the starting page first since it doesn't really give
use any good information that we can use for the next pass.

Fix a couple comment typos.

Rework the algorithm to allocate contiguous pages to be much much faster.
(read the comments if you want to know how it's done).

Sync with HEAD.

Commit is split, to avoid a "too many arguments" protocol error.

sync with head.

Sync with HEAD.

remove "#define PGALLOC_VERBOSE".

For this physical address printing use uintmax_t since on Xen PAE this length
(64) is not the same as the base architecture (32).

fix compile failure when PGALLOC_VERBOSE is defined.

Sync w/ -current. 34 merge conflicts to follow.

sync with head.

Sync with HEAD.

Also fix build.

- vm_page: put listq, pageq into a union alongside a LIST_ENTRY, so we can
  use both types of list.

- Make page coloring and idle zero state per-CPU.

- Maintain per-CPU page freelists. When freeing, put pages onto the local
  CPU's lists and the global lists. When allocating, prefer to take pages
  from the local CPU. If none are available take from the global list as
  done now. Proposed on tech-kern@.

sync with head

Sync with HEAD.

UVM_PAGEZERO_TARGET -> UVM_PAGEZERO_LOWAT

sync with head.

sync with head.

Remove clause 3 and 4 from TNF licenses

Sync with HEAD.

sync with head.

sync with HEAD

sync with head.

Assert uvm_fpageqlock is held in a few more places.

sync with head.

Sync with HEAD.

Sync with HEAD.

Merge unobtrusive locking changes from the vmlocking branch.

file uvm_pglist.c was added on branch matt-mips64 on 2007-07-21 19:21:56 +0000

Merge unobtrusive locking changes from the vmlocking branch.

Pull in the initial set of changes for the vmlocking branch.

- sync with head.
- move sched_changepri back to kern_synch.c as it doesn't know PPQ anymore.

sync with head.

Replace the Mach-derived boolean_t type with the C99 bool type.  A
future commit will replace use of TRUE and FALSE with true and false.

sync with head.

Sync with head.

merge yamt-pdpolicy branch.
	- separate page replacement policy from the rest of kernel
	- implement an alternative replacement policy

make UVM_KICK_PDAEMON() a real function and stop including
uvm_pdpolicy.h from uvm.h.  this also fixes build of pmap(1).

sync with head

sync with head.

Sync with head.

Merge 2006-05-24 NetBSD-current into the "peter-altq" branch.

sync with head.

XXX: GCC uninitialized.

separate page replacement policy from the rest of kernel.

merge ktrace-lwp.

Sync with HEAD. Here we go again...

Use ANSI function decls.

Fix the sync with head I botched.

Sync with HEAD.

make free page queue filo rather than fifo.
data in pages freed more recently are more likely on cpu cache.

Sync with HEAD

Drop trailing spaces.

add a DEBUG check if freed PG_ZERO pages are really zero-filled.

in uvm_pagefree and friends, if freed pages have been marked by
PG_ZERO flag, put them to PGFL_ZEROS queue rather than default one
so that we can re-use zero-filled pages efficiently.

use VM_PAGE_TO_PHYS macro instead of using phys_addr directly.

sync comments with reality

Make PGALLOC_VERBOSE compile where size_t != int.

Catch up to -current

/*CONTCOND*/ while (0)'ed macros

sync kqueue branch with HEAD

Catch up to -current.

catch up with -current.

Big cleanup and speed improvements to pglist_alloc code:
-pass vm_physseg* instead of physseg index, and PFN (int) instead
 of physical address (could be done even more)
-simplify detection of boundary crossing and behave more intelligently
 in this case
-take stuff out of the inner loops, or put into "#ifdef DEBUG"
 (because we move along physsegs we don't need to check that the
  pages are physically contigous)
-make the "simple" and "contigous" branches look more uniform; at
 least the outer loops might coalesce one day

catch up with -current on kqueue branch

catch up with -current.

Shift by PAGE_SHIFT instead of dividing by PAGE_SIZE.

Catch up to -current.

Make the DMA memory allocators (uvm_pglistalloc())
obey the preferences expressed by freelist assignment,
to avoid wasting valuable "low memory" to devices which
don't really need it.
comments:
-I'm not sure searching the physsegs within a freelist
 beginning with the biggest is the right thing. This is
 what the "memory steal" code in uvm_page.c does, so
 keep it consistent.
-There seems to be some confusion whether the upper
 address limit passed is inclusive or not. Stays on
 the save side, possibly leaving one page out.
-The boundary/pagemask check can be simplified, also some
 arguments passed are only used for diagnostic checks.
-Integration with UVM_PAGE_TRKOWN???

move initialization of the "struct pglist" returned by uvm_pglistalloc()
from the calling code into uvm_pglistalloc() itself for consistency
and easier error handling

Catch up with -current.

Add another allocator to uvm_pglistalloc() which is used in the case where
no alignment / boundary / nsegs restrictions apply.
This one doesn't insist in a contigous range, and it honours the "waitok"
flag, thus succeeds in situations which were hopeless with the existing one.

(A solution which searches for a minimum number of contiguous ranges using
some best-fit or so algorithm would be expensive to implement; I believe the
"either-or" done here does reflect the current use by bus_dma quite well.)

Now agp memory allocation is robust for me. (tested on i810)

Convert the fpageqlock to a spin mutex at IPL_VM and rename it
to fpageq_mutex.

Sync kqueue branch with -current.

Catch up to -current.

Sync the thorpej-mips-cache branch with -current.

add RCSIDs, and in some cases, slightly cleanup #include order

Catch up with -current.

Catch up to -current.

a whole bunch of changes to improve performance and robustness under load:

 - remove special treatment of pager_map mappings in pmaps.  this is
   required now, since I've removed the globals that expose the address range.
   pager_map now uses pmap_kenter_pa() instead of pmap_enter(), so there's
   no longer any need to special-case it.
 - eliminate struct uvm_vnode by moving its fields into struct vnode.
 - rewrite the pageout path.  the pager is now responsible for handling the
   high-level requests instead of only getting control after a bunch of work
   has already been done on its behalf.  this will allow us to UBCify LFS,
   which needs tighter control over its pages than other filesystems do.
   writing a page to disk no longer requires making it read-only, which
   allows us to write wired pages without causing all kinds of havoc.
 - use a new PG_PAGEOUT flag to indicate that a page should be freed
   on behalf of the pagedaemon when it's unlocked.  this flag is very similar
   to PG_RELEASED, but unlike PG_RELEASED, PG_PAGEOUT can be cleared if the
   pageout fails due to eg. an indirect-block buffer being locked.
   this allows us to remove the "version" field from struct vm_page,
   and together with shrinking "loan_count" from 32 bits to 16,
   struct vm_page is now 4 bytes smaller.
 - no longer use PG_RELEASED for swap-backed pages.  if the page is busy
   because it's being paged out, we can't release the swap slot to be
   reallocated until that write is complete, but unlike with vnodes we
   don't keep a count of in-progress writes so there's no good way to
   know when the write is done.  instead, when we need to free a busy
   swap-backed page, just sleep until we can get it busy ourselves.
 - implement a fast-path for extending writes which allows us to avoid
   zeroing new pages.  this substantially reduces cpu usage.
 - encapsulate the data used by the genfs code in a struct genfs_node,
   which must be the first element of the filesystem-specific vnode data
   for filesystems which use genfs_{get,put}pages().
 - eliminate many of the UVM pagerops, since they aren't needed anymore
   now that the pager "put" operation is a higher-level operation.
 - enhance the genfs code to allow NFS to use the genfs_{get,put}pages
   instead of a modified copy.
 - clean up struct vnode by removing all the fields that used to be used by
   the vfs_cluster.c code (which we don't use anymore with UBC).
 - remove kmem_object and mb_object since they were useless.
   instead of allocating pages to these objects, we now just allocate
   pages with no object.  such pages are mapped in the kernel until they
   are freed, so we can use the mapping to find the page to free it.
   this allows us to remove splvm() protection in several places.

The sum of all these changes improves write throughput on my
decstation 5000/200 to within 1% of the rate of NetBSD 1.5
and reduces the elapsed time for "make release" of a NetBSD 1.5
source tree on my 128MB pc to 10% less than a 1.5 kernel took.

Catch up with -current.

Macro'ize the code that checks the free and inactive thresholds and
wakes the pagedaemon.

Catch up to -current.

replace vm_page_t with struct vm_page *.

remove trailing whitespace.

Implement page coloring, using a round-robin bucket selection
algorithm (Solaris calls this "Bin Hopping").

This implementation currently relies on MD code to define a
constant defining the number of buckets.  This will change
reasonably soon (MD code will be able to dynamically size
the bucket array).

Sync with HEAD.

clean up DIAGNOSTIC checks, use KASSERT().

Sync with HEAD.

lots of cleanup:
use queue.h macros and KASSERT().
address amap offsets in pages instead of bytes.
make amap_ref() and amap_unref() take an amap, offset and length
  instead of a vm_map_entry_t.
improve whitespace and comments.

Update thorpej_scsipi to -current as of a month ago

remove include of <vm/vm.h>

Clean up a comment.

Changes necessary to implement pre-zero'ing of pages in the idle loop:
- Make page free lists have two actual queues: known-zero pages and
  pages with unknown contents.
- Implement uvm_pageidlezero().  This function attempts to zero up to
  the target number of pages until the target has been reached (currently
  target is `all free pages') or until whichqs becomes non-zero (indicating
  that a process is ready to run).
- Define a new hook for the pmap module for pre-zero'ing pages.  This is
  used to zero the pages using uncached access.  This allows us to zero
  as many pages as we want without polluting the cache.

In order to use this feature, each platform must add the appropropriate
glue in their idle loop.

Garbage collect thread_sleep()/thread_wakeup() left over from the old
Mach VM code.  Also nuke iprintf(), which was no longer used anywhere.

Add proclist locking where appropriate.

Sync w/ -current.

merge everything from chs-ubc branch.

- Change uvm_{lock,unlock}_fpageq() to return/take the previous interrupt
  level directly, instead of making the caller wrap the calls in
  splimp()/splx().
- Add a comment documenting that interrupts that cause memory allocation
  must be blocked while the free page queue is locked.

Since interrupts must be blocked while this lock is asserted, tying them
together like this helps to prevent mistakes.

in uvm_pglistalloc(), treat pages being paged out as "free"
when deciding whether to wakeup the pagedaemon.
also, don't unlock the free page queue until we've done the wakeup.

Merge paddr_t changes into the main branch.

Split vm_offset_t and vm_size_t into paddr_t, psize_t, vaddr_t, and vsize_t.

Add support for multiple memory free lists.  There is at least one
default free list, and 0 - N additional free list, in order of descending
priority.

A new page allocation function, uvm_pagealloc_strat(), has been added,
providing three page allocation strategies:

	- normal: high -> low priority free list walk, taking the
	  page off the first free list that has one.

	- only: attempt to allocate a page only from the specified free
	  list, failing if that free list has none available.

	- fallback: if `only' fails, fall back on `normal'.

uvm_pagealloc(...) is provided for normal use (and is a synonym for
uvm_pagealloc_strat(..., UVM_PGA_STRAT_NORMAL, 0); the free list argument
is ignored for the `normal' case).

uvm_page_physload() now specified which free list the pages will be
loaded onto.  This means that some platforms which have multiple physical
memory segments may define additional vm_physsegs if they wish to break
individual physical segments into differing priorities.

Machine-dependent code must define _at least_ the following constants
in <machine/vmparam.h>:

	VM_NFREELIST: the number of free lists the system will have

	VM_FREELIST_DEFAULT: the default freelist (should always be 0,
	but is defined in machdep code so that it's with all of the
	other free list-related constants).

Additional free list names may be defined by machine-dependent code, but
they will only be used by machine-dependent code (e.g. for loading the
vm_physsegs).

Remove inclusions of syscall (and syscall argument) related header files;
we don't need them here.

KNF.

RCS ID police.

initial import of the new virtual memory system, UVM, into -current.

UVM was written by chuck cranor <chuck@maria.wustl.edu>, with some
minor portions derived from the old Mach code.  i provided some help
getting swap and paging working, and other bug fixes/ideas.  chuck
silvers <chuq@chuq.com> also provided some other fixes.

this is the UVM kernel code portion.


this will be KNF'd shortly.  :-)

Initial revision