CVS log for src/libexec/lfs_cleanerd/lfs_cleanerd.8

Up to [cvs.NetBSD.org] / src / libexec / lfs_cleanerd

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

s/threshhold/threshold

Mostly merge changes from HEAD upto 20200411

Pull up following revision(s) (requested by brad in ticket #110):

	libexec/lfs_cleanerd/lfs_cleanerd.c: revision 1.59
	libexec/lfs_cleanerd/lfs_cleanerd.8: revision 1.19
	sbin/resize_lfs/resize_lfs.c: revision 1.15
	usr.sbin/puffs/rump_lfs/rump_lfs.c: revision 1.19
	libexec/lfs_cleanerd/lfs_cleanerd.c: revision 1.60
	lib/libutil/getdiskrawname.c: revision 1.6
	tests/fs/common/fstest_lfs.c: revision 1.7

Use getdiskrawname to find the device name.
Reviewed by Christos

 -

Teach getdiskrawname and getdiskcookedname about zvols.
Reviewed by Christos

 -

Add support for passing the raw device name separate from the
filesystem.  This is useful in the case where the cleaner is compiled
into code, such as rump_lfs and the ATF tests.  This helps to fix
bin/54488

 -

The cleaner is compiled into rump_lfs and executed as a thread.  Pass
in the raw device using the new -J option.  This avoids the use of
getdiskrawname which is not particularly rump safe in this context and
insures that the rump container device is used for cleaning, not the
outer device.

New sentence, new line. Punctuation improvements.

Add support for passing the raw device name separate from the
filesystem.  This is useful in the case where the cleaner is compiled
into code, such as rump_lfs and the ATF tests.  This helps to fix
bin/54488

Describe -D. Bump date.

Bump date for previous.

Temporarily disable -c (file coalescing) until data corruption issues can
be resolved.

Document -i option, update date; at the request of wiz@.

A new version of the cleaner.  In general, works about as well as the old
cleaner, but with more legible code.

Includes code for reading and writing to the raw disk device (so that an
unmounted fs could be cleaned), for the use of a single daemon to clean
multiple filesystems to save on resources, and for recording the old
contents of cleaned segments to offline storage for regression testing of
the LFS system as a whole; though these new features are not properly
tested at this point.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22284, verified by myself.

Begin new sentences on new lines.
Patch from Robert Elz (kre at munnari oz au).

update lfs_cleanerd manual page for new -c option

Whitespace nits, sort SEE ALSO.

Merge the short-lived perseant-lfsv2 branch into the trunk.

Kernels and tools understand both v1 and v2 filesystems; newfs_lfs
generates v2 by default.  Changes for the v2 layout include:

- Segments of non-PO2 size and arbitrary block offset, so these can be
  matched to convenient physical characteristics of the partition (e.g.,
  stripe or track size and offset).

- Address by fragment instead of by disk sector, paving the way for
  non-512-byte-sector devices.  In theory fragments can be as large
  as you like, though in reality they must be smaller than MAXBSIZE in size.

- Use serial number and filesystem identifier to ensure that roll-forward
  doesn't get old data and think it's new.  Roll-forward is enabled for
  v2 filesystems, though not for v1 filesystems by default.

- The inode free list is now a tailq, paving the way for undelete (undelete
  is not yet implemented, but can be without further non-backwards-compatible
  changes to disk structures).

- Inode atime information is kept in the Ifile, instead of on the inode;
  that is, the inode is never written *just* because atime was changed.
  Because of this the inodes remain near the file data on the disk, rather
  than wandering all over as the disk is read repeatedly.  This speeds up
  repeated reads by a small but noticeable amount.

Other changes of note include:

- The ifile written by newfs_lfs can now be of arbitrary length, it is no
  longer restricted to a single indirect block.

- Fixed an old bug where ctime was changed every time a vnode was created.
  I need to look more closely to make sure that the times are only updated
  during write(2) and friends, not after-the-fact during a segment write,
  and certainly not by the cleaner.

Import of what I've been calling "LFSv2", that is, LFS with some features
added that require changes to the on-disk data structures.  These include:

- 64-bit time in everything but inodes
- User-specified segment offset, and segment size no longer
  restricted to PO2.
- Serial number on segment summaries in addition to timestamp, and
  a new volume identifier, to make roll-forward feasible without
  fear of finding old data and thinking it was new.

Although I think this version works at least as well as what's on the trunk,
we're not done yet; hence this commit is going in on a branch and not on
the trunk.  Enhancements that are not here yet include fragment addressing,
like FFS does, instead of block addressing.

in SEE also, xref newfs_lfs(8), sort entries and seperate by ","

Pull up LFS userland changes to correspond with the previous commit's
kernel changes (approved by thorpej):

[basesrc/libexec/lfs_cleanerd:]
cleanerd.c, 1.20--1.22           (MIN_FREE_SEG -> lfs_minfreeseg ;
                                 read bfree/avail from CLEANERINFO ;
                                 changes to definition of -n and -b)
lfs_cleanerd.8, 1.7              (update man page to current behavior)
library.c, 1.16                  (fix comment)

[basesrc/sbin/fsck_lfs:]
pass5.c, 1.6                     (calculate/fix lfs_avail and lfs_bfree)

[basesrc/sbin/newfs_lfs:]
config.h, 1.2--1.5               (MINFREE=20, remove FFS cruft ;
                                 add DFL_MIN_FREE_SEGS=8 and MINFREE=10 ;
                                 set DFL_MIN_FREE_SEGS=10 ;
                                 set DFL_MIN_FREE_SEGS=20)
extern.h, 1.2                    (correct function declaration for make_lfs)
newfs.c, 1.4                     (add -M flag)
lfs.c, 1.13--1.14, 1.16--1.18    (change lfs_bfree initialization ;
                                 MIN_FREE_SEG -> lfs_minfreeseg ;
                                 only 10 superblocks and print nicely ;
                                 correct init calculation of
                                   lfs_bfree/lfs_avail to allow fs to
								   fill ;
                                 make -N dtrt)

[basesrc/sbin/mount_lfs:]
mount_lfs.8, 1.7                 (document -N, -b flags)
mount_lfs.c, 1.10                (default cleanerd to -b -n 4; add -N,
                                 -b flags)

[basesrc/usr.sbin/dumplfs:]
dumplfs.c, 1.15                  (print only the SEGUSEs specified with -s)

Various bug-fixes to LFS, to wit:


Kernel:

* Add runtime quantity lfs_ravail, the number of disk-blocks reserved
  for writing.  Writes to the filesystem first reserve a maximum amount
  of blocks before their write is allowed to proceed; after the blocks
  are allocated the reserved total is reduced by a corresponding amount.

  If the lfs_reserve function cannot immediately reserve the requested
  number of blocks, the inode is unlocked, and the thread sleeps until
  the cleaner has made enough space available for the blocks to be
  reserved.  In this way large files can be written to the filesystem
  (or, smaller files can be written to a nearly-full but thoroughly
  clean filesystem) and the cleaner can still function properly.

* Remove explicit switching on dlfs_minfreeseg from the kernel code; it
  is now merely a fs-creation parameter used to compute dlfs_avail and
  dlfs_bfree (and used by fsck_lfs(8) to check their accuracy).  Its
  former role is better assumed by a properly computed dlfs_avail.

* Bounds-check inode numbers submitted through lfs_bmapv and lfs_markv.
  This prevents a panic, but, if the cleaner is feeding the filesystem
  the wrong data, you are still in a world of hurt.

* Cleanup: remove explicit references of DEV_BSIZE in favor of
  btodb()/dbtob().

lfs_cleanerd:

* Make -n mean "send N segments' blocks through a single call to
  lfs_markv".  Previously it had meant "clean N segments though N calls
  to lfs_markv, before looking again to see if more need to be cleaned".
  The new behavior gives better packing of direct data on disk with as
  little metadata as possible, largely alleviating the problem that the
  cleaner can consume more disk through inefficient use of metadata than
  it frees by moving dirty data away from clean "holes" to produce
  entirely clean segments.

* Make -b mean "read as many segments as necessary to write N segments
  of dirty data back to disk", rather than its former meaning of "read
  as many segments as necessary to free N segments worth of space".  The
  new meaning, combined with the new -n behavior described above,
  further aids in cleaning storage efficiency as entire segments can be
  written at once, using as few blocks as possible for segment summaries
  and inode blocks.

* Make the cleaner take note of segments which could not be cleaned due
  to error, and not attempt to clean them until they are entirely free
  of dirty blocks.  This prevents the case in which a cleanerd running
  with -n 1 and without -b (formerly the default) would spin trying
  repeatedly to clean a corrupt segment, while the remaining space
  filled and deadlocked the filesystem.

* Update the lfs_cleanerd manual page to describe all the options,
  including the changes mentioned here (in particular, the -b and -n
  flags were previously undocumented).

fsck_lfs:

* Check, and optionally fix, lfs_avail (to an exact figure) and
  lfs_bfree (within a margin of error) in pass 5.

newfs_lfs:

* Reduce the default dlfs_minfreeseg to 1/20 of the total segments.

* Add a warning if the sgs disklabel field is 16 (the default for FFS'
  cpg, but not usually desirable for LFS' sgs: 5--8 is a better range).

* Change the calculation of lfs_avail and lfs_bfree, corresponding to
  the kernel changes mentioned above.

mount_lfs:

* Add -N and -b options to pass corresponding -n and -b options to
  lfs_cleanerd.

* Default to calling lfs_cleanerd with "-b -n 4".


[All of these changes were largely tested in the 1.5 branch, with the
idea that they (along with previous un-pulled-up work) could be applied
to the branch while it was still in ALPHA2; however my test system has
experienced corruption on another filesystem (/dev/console has gone
missing :^), and, while I believe this unrelated to the LFS changes, I
cannot with good conscience request that the changes be pulled up.]

More and more .Os cleanups.  .Os is defined in the tmac.doc-common file,
so we shouldn't override it with versions in the manpages.  Many more to
come.

macroize BSD, NetBSD, FreeBSD and misc cleanup

Fix usage of .Nm.

WARNS?=1.  RCS ids.

Fix broken uses of Dd.  Both the mdoc and mdoc.samples pages agree:
.Dd is supposed to be invoked like:
	.Dd month day, year
e.g. ".Dd January 25, 1989", rather than:
	.Dd "month day, year"
which is what these pages did.

Import original 4.4-Lite version.

From 4.4-Lite, with local changes.