CVS log for src/sys/uvm/uvm_object.h

Up to [cvs.NetBSD.org] / src / sys / uvm

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

fix various typos in comments.

centralize calls from UVM to radixtree into a few functions.
in those functions, assert that the object lock is held in
the correct mode.

Merge changes from current as of 20200406

Make uvm_pagemarkdirty() responsible for putting vnodes onto the syncer
work list.  Proposed on tech-kern@.

Sync with head.

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart.  v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap.  Others to follow later.

Sync with head.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
  pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
  precisely in uvm layer.

Merge from yamt-pagecache:

- do gang lookup of pages using radixtree.
- remove now unused uvm_object::uo_memq and vm_page::listq.queue.

Merge from yamt-pagecache: use radixtree for page lookup.

rbtree page lookup was introduced during the NetBSD 5.0 development cycle to
bypass lock contention problems with the (then) global page hash, and was a
temporary solution to allow us to make progress.  radixtree is the intended
replacement.

Ok yamt@.

Pull up following revision(s) (requested by rmind in ticket #694):
	sys/uvm/uvm_aobj.h: revision 1.22
	sys/uvm/uvm_aobj.c: revision 1.117
	sys/uvm/uvm_aobj.c: revision 1.118
	sys/uvm/uvm_aobj.c: revision 1.119
	sys/uvm/uvm_object.h: revision 1.33
- Describe uvm_aobj and the lock order.
- Remove unnecessary uao_dropswap_range1() wrapper.
- KNF.  Sprinkle some __cacheline_aligned.
- Manage anonymous UVM object reference count with atomic ops.
- Fix an old bug of possible lock against oneself (uao_detach_locked() is
  called from uao_swap_off() with uao_list_lock acquired).  Also removes
  the try-lock dance in uao_swap_off(), since the lock order changes.

Resync to 2012-11-19 00:00:00 UTC

sync with head

- Manage anonymous UVM object reference count with atomic ops.
- Fix an old bug of possible lock against oneself (uao_detach_locked() is
  called from uao_swap_off() with uao_list_lock acquired).  Also removes
  the try-lock dance in uao_swap_off(), since the lock order changes.

- fix integrity sync.
  putpages for integrity sync (fsync, msync with MS_SYNC, etc) should not
  skip pages being written back by other threads.

- adapt to radix tree tag api changes.

sync with head

merge to -current.

Describe UVM object and explain lock sharing a little.

remove pg->listq and uobj->memq

page cache related changes

- maintain object pages in radix tree rather than rb tree.
- reduce unnecessary page scan in putpages.  esp. when an object has a ton of
  pages cached but only a few of them are dirty.
- reduce the number of pmap operations by tracking page dirtiness more
  precisely in uvm layer.
- fix nfs commit range tracking.
- fix nfs write clustering.  XXX hack

Catchup with rmind-uvmplock merge.

Welcome to 5.99.53!  Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
  New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
  the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
  Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
  kernel-lock on some ports).  Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

Sync with HEAD.

sync with head

Sync with HEAD

udpate license clauses on my code to match the new-style BSD licenses.
based on diff that rmind@ sent me.

no functional change with this commit.

Include uvm/uvm_pglist.h for struct pglist.

Sync with HEAD (-D20101022).

sync with head

Rename rb.h to rbtree.h, as it is more appropriate (c.f. ptree.h).  Also
helps find code that hasn't been updated to use the new rbtree API.

Fixes/improvements to RB-tree implementation:
1. Fix inverted node order, so that negative value from comparison operator
   would represent lower (left) node, and positive - higher (right) node.
2. Add an argument (i.e. "context"), passed to comparison operators.
3. Change rb_tree_insert_node() to return a node - either inserted one or
   already existing one.
4. Amend the interface to manipulate the actual object, instead of the
   rb_node (in a similar way as Patricia-tree interface does).
5. Update all RB-tree users accordingly.

XXX: Perhaps rename rb.h to rbtree.h, since cleaning-up..

1-3 address the PR/43488 by Jeremy Huddleston.

Passes RB-tree regression tests.
Reviewed by: matt@, christos@

Don't expose uvm_page.h internal for usual uvm(9) users.

Add ubc_purge() and purge/deassociate any related UBC entries during
object (usually, vnode) destruction.  Since locking (and thus object)
is required to enter/remove mappings - object is not allowed anymore
to disappear with any UBC entries left.

From original patch by ad@ with some modifications.

Change struct uvm_object::vmobjlock to be dynamically allocated with
mutex_obj_alloc().  It allows us to share the locks among UVM objects.

sync with head.

Sync w/ -current. 34 merge conflicts to follow.

sync with head.

Sync with HEAD.

Also fix build.

Replace the global vm_page hash with a per vm_object rbtree.
Proposed on tech-kern@.

sync with head

Use atomics to maintain v_usecount.

Sync with HEAD.

sync with head

sync with HEAD

Sync with HEAD

Merge vmlocking2 to head.

Sync with HEAD.

sync with head

Pull the vmlocking changes into a new branch.

Sync with HEAD.

constify pagerops.

destroy vmobjlock.

Pull in the initial set of changes for the vmlocking branch.

sync with head.

Sync with head.

sync with head

move some knowledge about vnode into uvm_vnode.c.

sync with head.

merge ktrace-lwp.

Sync with HEAD. Here we go again...

Pull up following revision(s) (requested by yamt in ticket #688):
	sys/miscfs/genfs/genfs_vnops.c: revision 1.98 via patch
	sys/ufs/ffs/ffs_vfsops.c: revision 1.165
	sys/ufs/lfs/lfs_extern.h: revision 1.69
	sys/fs/filecorefs/filecore_vfsops.c: revision 1.20
	sys/nfs/nfs_node.c: revision 1.80
	sys/fs/smbfs/smbfs_node.c: revision 1.24
	sys/fs/cd9660/cd9660_vfsops.c: revision 1.24
	sys/fs/msdosfs/msdosfs_denode.c: revision 1.8
	sys/miscfs/genfs/genfs_node.h: revision 1.6
	sys/ufs/lfs/lfs_vfsops.c: revision 1.183
	sys/ufs/ext2fs/ext2fs_vfsops.c: revision 1.86
	sys/fs/adosfs/advfsops.c: revision 1.23
	sys/fs/ntfs/ntfs_vfsops.c: revision 1.31
- constify genfs_ops.
- use member designators.

        sys/miscfs/genfs/genfs_vnops.c: revision 1.99 via patch
genfs_getpages: don't forget to put the vnode onto the syncer's work que
ue
even in the case of PGO_LOCKED.

        sys/uvm/uvm_bio.c: revision 1.40
        sys/uvm/uvm_pager.h: revision 1.29
        sys/miscfs/genfs/genfs_vnops.c: revision 1.100 via patch
        sys/ufs/ufs/ufs_inode.c: revision 1.50
- introduce PGO_NOBLOCKALLOC and use it for ubc mapping
  to prevent unnecessary block allocations in the case that
  page size > block size.
- ufs_balloc_range: use VM_PROT_WRITE+PGO_NOBLOCKALLOC rather than
  VM_PROT_READ.

        sys/uvm/uvm_fault.c: revision 1.96
        sys/miscfs/genfs/genfs_vnops.c: revision 1.101 via patch
        sys/uvm/uvm_object.h: revision 1.19
        sys/miscfs/genfs/genfs_node.h: revision 1.7
ensure that vnodes with dirty pages are always on syncer's queue.
- genfs_putpages: wait for i/o completion of PG_RELEASED/PG_PAGEOUT pages by
  setting "wasclean" false when encountering them.
  suggested by Stephan Uphoff in PR/24596 (1).
- genfs_putpages: write protect pages when cleaning out, if
  we're going to take the vnode off the syncer's queue.
  uvm_fault: don't write-map pages unless its vnode is already on
  the syncer's queue.
  fix PR/24596 (3) but in the different way from the suggested fix.
  (to keep our current behaviour, ie. not to require explicit msync.
  discussed on tech-kern@.)
- genfs_putpages: don't mistakenly take a vnode off the queue
  by introducing a generation number in genfs_node.
  genfs_getpages: increment the generation number.
  suggested by Stephan Uphoff in PR/24596 (2).
- add some assertions.

        sys/miscfs/genfs/genfs_vnops.c: revision 1.102 via patch
genfs_putpages: don't bother to clean the vnode unless VONWORKLST.

        sys/ufs/ffs/ffs_vnops.c: revision 1.71
ffs_full_fsync: because VBLK/VCHR can be mmap'ed,
do VOP_PUTPAGES for them as well.

        sys/uvm/uvm_fault.c: revision 1.97
uvm_fault: check a correct object in the case of layered filesystems.
fix PR/30811 from Jukka Salmi.

        sys/uvm/uvm_object.h: revision 1.20
        sys/ufs/ffs/ffs_vfsops.c: revision 1.167
        sys/uvm/uvm_bio.c: revision 1.41
        sys/ufs/ufs/ufs_vnops.c: revision 1.129
        sys/uvm/uvm_mmap.c: revision 1.92
        sys/uvm/uvm_fault.c: revision 1.98
        sys/kern/vfs_subr.c: revision 1.252
        sys/fs/msdosfs/denode.h: revision 1.5
        sys/miscfs/genfs/genfs_vnops.c: revision 1.103 via patch
        sys/fs/msdosfs/msdosfs_denode.c: revision 1.9
        sys/sys/vnode.h: revision 1.141
        sys/ufs/ufs/ufs_inode.c: revision 1.51
        sys/ufs/ufs/ufs_extern.h: revision 1.45 via patch
        sys/miscfs/genfs/genfs_node.h: revision 1.8
        sys/ufs/lfs/lfs_vfsops.c: revision 1.184
        sys/uvm/uvm_pager.h: revision 1.30
        sys/ufs/ext2fs/ext2fs_vfsops.c: revision 1.87
update file timestamps for nfsd loaned-read and mmap.
PR/25279.  discussed on tech-kern@.

        sys/miscfs/genfs/genfs_vnops.c: revision 1.104 via patch
don't write-protect wired pages.  pointed by Chuck Silvers.
for now, leave a vnode on the syncer's queue, as suggested by him.

        sys/ufs/ffs/ffs_vnops.c: revision 1.72
revert VCHR part of ffs_vnops.c 1.71.
as VCHR uses the device pager, no point to call VOP_PUTPAGES here.
pointed by Chuck Silvers.

update file timestamps for nfsd loaned-read and mmap.
PR/25279.  discussed on tech-kern@.

ensure that vnodes with dirty pages are always on syncer's queue.

- genfs_putpages: wait for i/o completion of PG_RELEASED/PG_PAGEOUT pages by
  setting "wasclean" false when encountering them.
  suggested by Stephan Uphoff in PR/24596 (1).

- genfs_putpages: write protect pages when cleaning out, if
  we're going to take the vnode off the syncer's queue.
  uvm_fault: don't write-map pages unless its vnode is already on
  the syncer's queue.

  fix PR/24596 (3) but in the different way from the suggested fix.
  (to keep our current behaviour, ie. not to require explicit msync.
  discussed on tech-kern@.)

- genfs_putpages: don't mistakenly take a vnode off the queue
  by introducing a generation number in genfs_node.
  genfs_getpages: increment the generation number.
  suggested by Stephan Uphoff in PR/24596 (2).

- add some assertions.

introduce a macro to initialize uvm_object and use it.

Fix the sync with head I botched.

Sync with HEAD.

Sync with HEAD

mincore: don't treat an aobj as a device mapping.

sync kqueue branch with HEAD

Catch up to -current.

catch up with -current.

catch up with -current on kqueue branch

Pull up revision 1.16 (requested by chs in ticket #329):
count aobj pages (most notably kernel stack pages) as anon pages
for memory usage-balancing purposes.

count aobj pages (most notably kernel stack pages) as anon pages
for memory usage-balancing purposes.

Catch up to -current.

When core dumping a process, don't dump maps backed up by the device pager.
(move the pagerops externs to uvm_object.h and out the C files).

Sync kqueue branch with -current.

Catch up to -current.

Sync the thorpej-mips-cache branch with -current.

- Add a new vnode flag VEXECMAP, which indicates that a vnode has
  executable mappings.  Stop overloading VTEXT for this purpose (VTEXT
  also has another meaning).
- Rename vn_marktext() to vn_markexec(), and use it when executable
  mappings of a vnode are established.
- In places where we want to set VTEXT, set it in v_flag directly, rather
  than making a function call to do this (it no longer makes sense to
  use a function call, since we no longer overload VTEXT with VEXECMAP's
  meaning).

VEXECMAP suggested by Chuq Silvers.

Catch up with -current.

Catch up to -current.

a whole bunch of changes to improve performance and robustness under load:

 - remove special treatment of pager_map mappings in pmaps.  this is
   required now, since I've removed the globals that expose the address range.
   pager_map now uses pmap_kenter_pa() instead of pmap_enter(), so there's
   no longer any need to special-case it.
 - eliminate struct uvm_vnode by moving its fields into struct vnode.
 - rewrite the pageout path.  the pager is now responsible for handling the
   high-level requests instead of only getting control after a bunch of work
   has already been done on its behalf.  this will allow us to UBCify LFS,
   which needs tighter control over its pages than other filesystems do.
   writing a page to disk no longer requires making it read-only, which
   allows us to write wired pages without causing all kinds of havoc.
 - use a new PG_PAGEOUT flag to indicate that a page should be freed
   on behalf of the pagedaemon when it's unlocked.  this flag is very similar
   to PG_RELEASED, but unlike PG_RELEASED, PG_PAGEOUT can be cleared if the
   pageout fails due to eg. an indirect-block buffer being locked.
   this allows us to remove the "version" field from struct vm_page,
   and together with shrinking "loan_count" from 32 bits to 16,
   struct vm_page is now 4 bytes smaller.
 - no longer use PG_RELEASED for swap-backed pages.  if the page is busy
   because it's being paged out, we can't release the swap slot to be
   reallocated until that write is complete, but unlike with vnodes we
   don't keep a count of in-progress writes so there's no good way to
   know when the write is done.  instead, when we need to free a busy
   swap-backed page, just sleep until we can get it busy ourselves.
 - implement a fast-path for extending writes which allows us to avoid
   zeroing new pages.  this substantially reduces cpu usage.
 - encapsulate the data used by the genfs code in a struct genfs_node,
   which must be the first element of the filesystem-specific vnode data
   for filesystems which use genfs_{get,put}pages().
 - eliminate many of the UVM pagerops, since they aren't needed anymore
   now that the pager "put" operation is a higher-level operation.
 - enhance the genfs code to allow NFS to use the genfs_{get,put}pages
   instead of a modified copy.
 - clean up struct vnode by removing all the fields that used to be used by
   the vfs_cluster.c code (which we don't use anymore with UBC).
 - remove kmem_object and mb_object since they were useless.
   instead of allocating pages to these objects, we now just allocate
   pages with no object.  such pages are mapped in the kernel until they
   are freed, so we can use the mapping to find the page to free it.
   this allows us to remove splvm() protection in several places.

The sum of all these changes improves write throughput on my
decstation 5000/200 to within 1% of the rate of NetBSD 1.5
and reduces the elapsed time for "make release" of a NetBSD 1.5
source tree on my 128MB pc to 10% less than a 1.5 kernel took.

Catch up to -current.

replace {simple_,}lock{_data,}_t with struct {simple,}lock {,*}.

Catch up with -current.

Sync with HEAD.

add UBC memory-usage balancing.  we track the number of pages in use for
each of the basic types (anonymous data, executable image, cached files)
and prevent the pagedaemon from reusing a given page if that would reduce
the count of that type of page below a sysctl-setable minimum threshold.
the thresholds are controlled via three new sysctl tunables:
vm.anonmin, vm.vnodemin, and vm.vtextmin.  these tunables are the
percentages of pageable memory reserved for each usage, and we do not allow
the sum of the minimums to be more than 95% so that there's always some
memory that can be reused.

Sync with HEAD.

Put the extern decl of uvm_vnodeops in uvm_object.h

Define a UVM_OBJ_IS_VNODE() macro to test if an object is a vnode.

Sync w/ -current.

Define a new kernel object type, "intrsafe", which are used for objects
which can be used in an interrupt context.  Use pmap_kenter*() and
pmap_kremove() only for mappings owned by these objects.

Fixes some locking protocol issues related to MP support, and eliminates
all of the pmap_enter vs. pmap_kremove inconsistencies.

Macro'ize the test for "object is a kernel object".

remove now >1 year old pre-release message.

KNF.

add/cleanup multiple inclusion protection.

restore rcsids

RCS ID police.

initial import of the new virtual memory system, UVM, into -current.

UVM was written by chuck cranor <chuck@maria.wustl.edu>, with some
minor portions derived from the old Mach code.  i provided some help
getting swap and paging working, and other bug fixes/ideas.  chuck
silvers <chuq@chuq.com> also provided some other fixes.

this is the UVM kernel code portion.


this will be KNF'd shortly.  :-)

Initial revision