CVS log for src/sbin/fsck_lfs/inode.c

Up to [cvs.NetBSD.org] / src / sbin / fsck_lfs

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

Merge changes from current as of 20200406

Avoid common symbols for fsck_lfs.

Pull up following revision(s) (requested by maya in ticket #330):
	sbin/fsck_lfs/inode.c: 1.69
	sbin/fsck_lfs/lfs.c: 1.73
	sbin/fsck_lfs/pass6.c: 1.50
	sbin/fsck_lfs/segwrite.c: 1.46
	sys/ufs/lfs/lfs.h: 1.202-1.203
	sys/ufs/lfs/lfs_accessors.h: 1.48
	sys/ufs/lfs/lfs_alloc.c: 1.136-1.137
	sys/ufs/lfs/lfs_balloc.c: 1.94
	sys/ufs/lfs/lfs_bio.c: 1.141
	sys/ufs/lfs/lfs_extern.h: 1.113
	sys/ufs/lfs/lfs_inode.c: 1.156-1.157
	sys/ufs/lfs/lfs_inode.h: 1.20, 1.21, 1.23
	sys/ufs/lfs/lfs_itimes.c: 1.20
	sys/ufs/lfs/lfs_pages.c: 1.13-1.15
	sys/ufs/lfs/lfs_rename.c: 1.22
	sys/ufs/lfs/lfs_segment.c: 1.270-1.275
	sys/ufs/lfs/lfs_subr.c: 1.94-1.97
	sys/ufs/lfs/lfs_syscalls.c: 1.175
	sys/ufs/lfs/lfs_vfsops.c: 1.360
	sys/ufs/lfs/lfs_vnops.c: 1.316-1.321
	sys/ufs/lfs/ulfs_inode.c: 1.20
	sys/ufs/lfs/ulfs_inode.h: 1.24
	sys/ufs/lfs/ulfs_lookup.c: 1.41
	sys/ufs/lfs/ulfs_quota2.c: 1.31
	sys/ufs/lfs/ulfs_readwrite.c: 1.24
	sys/ufs/lfs/ulfs_vnops.c: 1.49-1.50
Update inode member i_flag --> i_state to keep up with kernel changes
Move definition of IN_ALLMOD near the flag it's a mask for.
Now we can see that it doesn't match all the flags, but changing that will
require more careful thought.
Correct confusion between i_flag and i_flags
These will have to be renamed.
Spotted by Riastradh, thanks!
Add an XXX about the missing flags so it's not buried in a commit
message.
now the XXX count for LFS is 260
Rename i_flag to i_state.
The similarity to i_flags has previously caused errors.
Use continue to denote the no-op loop to match netbsd style
newline for extra clarity.
It isn't safe to drain dirops with seglock held, it'll deadlock if there
are any dirops. drain before grabbing seglock.
lfs_dirops == 0 is always true (as we already drained dirops), so omit
that part of the comparison.
Fixes a lot of LFS deadlocks. PR kern/52301
Many thanks to dholland for help analyzing coredumps
Ifdef out KDASSERT which fires on my machine.
Deduplicate sanity check that seglock is held on segunlock
Revert r1.272 fix to PR kern/52301, the performance hit is making things
unusable.
change lfs_nextsegsleep and lfs_allclean_wakeup to use condvar
XXX had to use lfs_lock in lfs_segwait, removed kernel_lock, is this
appropriate?
fix buffer overflow/KASSERT when cookies are supplied
lfs no longer uses the ffs-style struct direct, use the correct minimum
size
from dholland
XXX more wrong
Consistently use {,UN}MARK_VNODE macros rather than function calls.
Not much point doing anything after a panic call
Ask some question about the code in a XXX comment
XXX question our double-flushing of dirops
Fix typo in comment

Update inode member i_flag --> i_state to keep up with kernel changes

Add 64-bit directory entry structures, and adjust accessors accordingly.

The LFS64 directory entry has a 64-bit inode number. This is stored as
two 32-bit values to avoid inducing 64-bit alignment requirements.

The exposed type for manipulating directory entries is now
LFS_DIRHEADER, following the same convention as e.g. IFILE and SEGUSE.
(But with LFS_ on it, because.)

Pass around struct lfs_dirheader instead of struct lfs_direct.

Add an accessor function for directory names.

Tidyups/fixes preparatory to making d_name[] in struct lfs_direct size
0 instead of size LFS_MAXNAMLEN+1, and preparatory to having accessor
functions for d_name. In particular, don't create prototype entries
and copy them, and access the name field only for directory structures
that are in buffers with space for the name to exist.

Add and use accessor functions for more of the directory entry fields.

Add new accessors for the d_type and d_namlen fields of struct lfs_direct.
Napalm the old byteswap access logic for these.

Use daddr_t, not ulfs_daddr_t, as the latter's 32 bits wide.
Don't use either for on-disk items.
Declare external data in header files.
Part 3 of 3.

Use the lfs dinode accessors in place of the ufs-derived ones.
(Mostly.)

The ufs-derived ones are fake structure member macros, which are gross
and not very safe. Also, it seems that a lot of places in the lfs code
were using the ffsv1 branch of them unconditionally, and this way it's
guaranteed all those places have been updated.

Found while doing this: for non-devices, have getattr produce NODEV
in the rdev field instead of leaking the address of the first direct
block.

Part two of dinodes; use the same union everywhere.
(previously the ufs-derived code had things set up slightly different)

Remove a bunch of associated mess.

Hack up dinode usage to be 64 vs. 32 as needed. Part 1.

(This part changes the native lfs code; the ufs-derived code already
has 64 vs. 32 logic, but as aspects of it are unsafe, and don't
entirely interoperate cleanly with the lfs 64/32 stuff, pass 2 will be
rehashing that.)

Add IFILE32 and IFILE64 structures for the on-disk ifile entries.
Add and use accessors. There are also a bunch of places that cast and
I hope I've found them all...

Add a new lfs header file: lfs_accessors.h.

This contains all the accessor functions and macros out of lfs.h.
Add an include of lfs_accessors.h after all uses of lfs.h... except
for code that wants to define its own struct lfs-alike that the
accessors are supposed to play along with. For these, set STRUCT_LFS
and include lfs_accessors.h after the necessary structure has been
defined, so that lfs_accessors.h can emit functions in terms of it.

More lfs superblock accessors.
(This changes the rest of the code over; all the accessors were
already added.)

The difference between this commit and the previous one is arbitrary,
but the previous one passed the regression tests on its own so I'm
keeping it separate to help with any bisections that might be needed
in the future.

Switch to accessor functions for elements of the LFS on-disk
superblock. This will allow switching between 32/64 bit forms on the
fly; it will also allow handling LFS_EI reasonably tidily. (That
currently doesn't work on the superblock.)

It also gets rid of cpp abuse in the form of fake structure member
macros.

Also, instead of doing sleep/wakeup on &lfs_avail and &lfs_nextseg
inside the on-disk superblock, add extra elements to the in-memory
struct lfs for this. (XXX: these should be changed to condvars, but
not right now)

XXX: this migrates a structure needed by the lfs code in libsa (struct
salfs) into lfs.h, where it doesn't belong, but for the time being
this is necessary in order to allow the accessors (and the various
lfs macros and other goop that relies on them) to compile.

fix error messages containing \n

- Fix fallout for recent bread() change (removing cred arg).

Rebase to HEAD as of a few days ago.

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs.  ("Protocol error: too many arguments")

don't use sprintf

resync from head

Prefix most of the cpp macros with lfs_ and LFS_ to avoid conflicts with ffs.
This was done so that boot blocks that want to compile both FFS and LFS in
the same file work.

Tidy up the LFS userland build hacks.
Don't use -I${NETBSDSRCDIR}/sys; don't include files other than the
exported LFS headers, which are lfs.h, lfs_inode.h, and (for now)
lfs_extern.h.

DIRBLKSIZ -> LFS_DIRBLKSIZ
DIRECTSIZ -> LFS_DIRECTSIZ
DIRSIZ -> LFS_DIRSIZ
OLDDIRFMT -> LFS_OLDDIRFMT
NEWDIRFMT -> LFS_NEWDIRFMT
IFTODT -> LFS_IFTODT
DTTOIF -> LFS_DTTOIF

struct direct -> struct lfs_direct
struct dirtemplate -> struct lfs_dirtemplate
struct odirtemplate -> struct lfs_odirtemplate
DT_* -> LFS_DT_*

Stick LFS_ in front of IFMT, IFIFO, IFREG, etc. so as not to conflict
with the UFS copies of these symbols. (Which themselves ought to have
UFS_ stuck on.)

Move the dinode (on-disk inode) structures to lfs.h, since they are
and will be obviously required by userland tools that need to read
the on-disk structures.

Also, DINODE{1,2}_SIZE -> LFS_DINODE{1,2}_SIZE.

Cleanups and hacks to make lfs userland stuff build:
   - lfs_cksum.c doesn't actually need ulfs_inode.h any more.
   - neither does lfs_itimes.c.
   - add hacks to fsck_lfs to make it compile.
   - add hacks to newfs_lfs to make it compile.
   - fix warning in ulfs_quota.c when quotas are fully disabled
     (as I guess is happening with the rumpity version)

XXX: This commit adds -I${NETBSDSRCDIR}/sys to the Makefiles for
XXX: fsck_lfs, newfs_lfs, and lfs_cleanerd. This needs to be cleaned
XXX: up ASAP; but I consider this less problematic in the short term
XXX: than spewing ulfs_*.h into /usr/include.

ufs -> ulfs for fsck_lfs.

resync with head

sync with head

Stuff UFS_ in front of a few of ufs's symbols to reduce namespace
pollution. Specifically:
   ROOTINO -> UFS_ROOTINO
   WINO -> UFS_WINO
   NXADDR -> UFS_NXADDR
   NDADDR -> UFS_NDADDR
   NIADDR -> UFS_NIADDR
   MAXSYMLINKLEN -> UFS_MAXSYMLINKLEN
   MAXSYMLINKLEN_UFS[12] -> UFS[12]_MAXSYMLINKLEN (for consistency)

Sort out ext2fs's misuse of NDADDR and NIADDR; fortunately, these have
the same values in ext2fs and ffs.

No functional change intended.

Three changes in a single commit.

- drop the notion of frags (LFS fragments) vs fsb (FFS fragments)
  The code uses a complicated unity function that just makes the
  code difficult to understand.

- support larger sector sizes. Fix disk address computations
  to use DEV_BSIZE in the kernel as required by device drivers
  and to use sector sizes in userland.

- Fix several locking bugs in lfs_bio.c and lfs_subr.c.

Centralize time printing and deal with ctime possibly returning NULL.

Pull up following revision(s) (requested by manu in ticket #1288):
	sbin/fsck_lfs/inode.c: revision 1.40 via patch
	sbin/fsck_ffs/fsck_ffs.8: revision 1.44 via patch
	sbin/fsck_ext2fs/fsck.h: revision 1.14 via patch
	sbin/fsck_ext2fs/main.c: revision 1.30 via patch
	sbin/fsck_ffs/inode.c: revision 1.61 via patch
	sbin/fsck_ffs/main.c: revision 1.71 via patch
	sbin/fsck_ext2fs/inode.c: revision 1.23 via patch
	sbin/fsck_lfs/fsck.h: revision 1.18 via patch
	sbin/fsck_lfs/fsck_lfs.8: revision 1.21 via patch
	sbin/fsck_lfs/main.c: revision 1.38 via patch
	sbin/fsck_ext2fs/fsck_ext2fs.8: revision 1.15 via patch
	sbin/fsck_ffs/fsck.h: revision 1.47 via patch
Disable userid to username lookups by default. Add a -U flag to perform them.
In single user mode lookups that involve the network might not work and they
slow down fsck.

Sync with HEAD.

Disable userid to username lookups by default. Add a -U flag to perform them.
In single user mode lookups that involve the network might not work and they
slow down fsck.

Sync w/ -current. 34 merge conflicts to follow.

Sync with HEAD.

sync with head.

Make sure all cached buffers with valid, not yet written data have been
run through copy-on-write.  Call fscow_run() with valid data where possible.

The LP_UFSCOW hack is no longer needed to protect ffs_copyonwrite() against
endless recursion.

- Add a flag B_MODIFY to bread(), breada() and breadn().  If set the caller
  intends to modify the buffer returned.

- Always run copy-on-write on buffers returned from ffs_balloc().

- Add new function ffs_getblk() that gets a buffer, assigns a new blkno,
  may clear the buffer and runs copy-on-write.  Process possible errors
  from getblk() or fscow_run().  Part of PR kern/38664.

Welcome to 4.99.63

Reviewed by: YAMAMOTO Takashi <yamt@netbsd.org>

Remove clause 3 and 4 from TNF licenses

sync with HEAD

Give brelse() a second argument so that it matches the kernel.
fsck_lfs now compiles again.

Fix malloc/realloc/calloc issues: always check and exit, use EEXIT instead
of 8.

Several fixes to improve the reliability of the roll-forward agent.
Also, note "properly orphaned" files as distinct from corrupted files.

On-disk inode accounting fix for roll-forward.

Don't doubly free an inode remove_ino() during roll-forward.

Various improvements to fsck_lfs, to wit:

* Add lfs_balloc capability to the lfs library.
* Extend the Ifile if we run out of free inodes when creating lost+found.
* Don't roll forward if we have allocated a lost+found, to avoid
  conflicts when adding new files in roll-forward.
* Make some messages slightly more verbose (e.g. include inode number,
  and use pwarn() instead of printf() so the messages include the device
  name when preening).
* Change superblock detection/avoidance to use the offset table in the
  primary superblock, rather than looking at the contents.
* Be more verbose about various operations when passed the -d flag,
  especially roll-forward.
* Be more careful about dirops during roll forward, since the cleaner can
  sometimes write blocks from dirop vnodes.  Detect and avoid this problem.
* Always check the free list, even if given -i; if we're going to write
  it we have to check it first.
* Mark inodes dirty when blocks are found during roll forward, so the
  inodes are written with the new block locations.
* Update size of inodes if blocks beyond EOF are found during roll
  forward.
* Fix segment accounting for blocks and inodes found during roll
  forward.
* Report statistics on roll forward: how many new/deleted/moved files
  and how many updated blocks (or "nothing new").
* Don't care if the device being checked is really a device, if we have
  been passed the -f flag (to facilitate automated testing).
* When writing to the disk, use the current time in the segment headers
  rathern than time 0.
* When passed the -i flag, locate the partial segment containing the
  Ifile inode and use that to calculate lfs_offset, lfs_curseg,
  lfs_nextseg.  (Again for automated testing.)

rename lfs.h to lfs_user.h so that it does not conflict.

64 bit inode changes

more const.

constify

Use the correct method to create a new inode, when we allocate lost+found.

Correct uninitialized variable issues in pass6.c and dir.c (PR#30411 and
PR#30394, respectively).

Apply patch (requested by perseant in ticket #242):
* fsck_lfs buffer cache fixes, including PR #29151
* Change fsck_lfs phase 0 message to reflect reality
* fsck_lfs: check phase 5 (cleanerinfo accounting) even on
  roll-forward
* Keep better track of the free list during roll-forward, avoiding
  a core dump
* Improve hash table use for fsck_lfs buffer and vnode cache
* Document fsck_lfs flag -f, and implement -q
* Add resize_lfs, including kernel support
* Add LFS to mountd's list of exportable filesystem types
* Make the LFS lkm work again [christos@]
* Add MP locking to the LFS kernel subsystem
* Fix pager_map deadlock in lfs_putpages()
* Avoid incomplete file extension that looks like "partial
  truncation" to fsck
* Use lfs_malloc for cleaner malloc, since the cleaner often runs
  in low-memory conditions.
* Use splay trees, not hash table, to track page allocation for
  write.
* Fix mkdir panic on full fs
* Fix page accounting leak by counting differently.
* Use rightly named structure for lfs_getattr [skrll@]
* Cosmetic changes for readability.

"#define lfs_devvp lfs_unlockvp" for readability, since that's what we
use it for in fsck_lfs/newfs_lfs.

ANSIfy a function declaration, remove obsolete "register" declarations.

zero-out dinode is not a proper way to 'clear' an lfs inode.

change KES license to standard NetBSD license

make this work for big (ie. with indirect blocks) directories
without spurious 'EXTRA ..' errors.

realloc pedant

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22308, verified by myself.

use bounded string op

in ckinode(), use idesc->id_number instead of dino.di_u.inumber
since the latter is invalid for pass2 and vget can return invalid vnode
for inode number 0.

Add support for UFS2. UFS2 is an enhanced FFS, adding support for
64 bit block pointers, extended attribute storage, and a few
other things.

This commit does not yet include the code to manipulate the extended
storage (for e.g. ACLs), this will be done later.

Originally written by Kirk McKusick and Network Associates Laboratories for
FreeBSD.

Add working writing ability to fsck_lfs, including roll-forward, based on
a partial-segment writer ported from the kernel.

Add code to UBCify LFS.  This is still behind "#ifdef LFS_UBC" for now
(there are still some details to work out) but expect that to go
away soon.  To support these basic changes (creation of lfs_putpages,
lfs_gop_write, mods to lfs_balloc) several other changes were made, to
wit:

* Create a writer daemon kernel thread whose purpose is to handle page
  writes for the pagedaemon, but which also takes over some of the
  functions of lfs_check().  This thread is started the first time an
  LFS is mounted.

* Add a "flags" parameter to GOP_SIZE.  Current values are
  GOP_SIZE_READ, meaning that the call should return the size of the
  in-core version of the file, and GOP_SIZE_WRITE, meaning that it
  should return the on-disk size.  One of GOP_SIZE_READ or
  GOP_SIZE_WRITE must be specified.

* Instead of using malloc(...M_WAITOK) for everything, reserve enough
  resources to get by and use malloc(...M_NOWAIT), using the reserves if
  necessary.  Use the pool subsystem for structures small enough that
  this is feasible.  This also obsoletes LFS_THROTTLE.

And a few that are not strictly necessary:

* Moves the LFS inode extensions off onto a separately allocated
  structure; getting closer to LFS as an LKM.  "Welcome to 1.6O."

* Unified GOP_ALLOC between FFS and LFS.

* Update LFS copyright headers to correct values.

* Actually cast to unsigned in lfs_shellsort, like the comment says.

* Keep track of which segments were empty before the previous
  checkpoint; any segments that pass two checkpoints both dirty and
  empty can be summarily cleaned.  Do this.  Right now lfs_segclean
  still works, but this should be turned into an effectless
  compatibility syscall.

Bump daddr_t to 64 bits. Replace it with int32_t in all places where
it was used on-disk, so that on-disk formats remain the same.
Remove ufs_daddr_t and ufs_lbn_t for the time being.

Pull up revision 1.13 (requested by perseant in ticket #131):
Re-checksum the superblock whenever it is marked dirty.
Tested on alpha.

Re-checksum the superblock whenever it is marked dirty.

Tested on alpha.

Add some \n to error messages.

Merge the short-lived perseant-lfsv2 branch into the trunk.

Kernels and tools understand both v1 and v2 filesystems; newfs_lfs
generates v2 by default.  Changes for the v2 layout include:

- Segments of non-PO2 size and arbitrary block offset, so these can be
  matched to convenient physical characteristics of the partition (e.g.,
  stripe or track size and offset).

- Address by fragment instead of by disk sector, paving the way for
  non-512-byte-sector devices.  In theory fragments can be as large
  as you like, though in reality they must be smaller than MAXBSIZE in size.

- Use serial number and filesystem identifier to ensure that roll-forward
  doesn't get old data and think it's new.  Roll-forward is enabled for
  v2 filesystems, though not for v1 filesystems by default.

- The inode free list is now a tailq, paving the way for undelete (undelete
  is not yet implemented, but can be without further non-backwards-compatible
  changes to disk structures).

- Inode atime information is kept in the Ifile, instead of on the inode;
  that is, the inode is never written *just* because atime was changed.
  Because of this the inodes remain near the file data on the disk, rather
  than wandering all over as the disk is read repeatedly.  This speeds up
  repeated reads by a small but noticeable amount.

Other changes of note include:

- The ifile written by newfs_lfs can now be of arbitrary length, it is no
  longer restricted to a single indirect block.

- Fixed an old bug where ctime was changed every time a vnode was created.
  I need to look more closely to make sure that the times are only updated
  during write(2) and friends, not after-the-fact during a segment write,
  and certainly not by the cleaner.

Change disk addressing unit to be the fragment, instead of the disk sector.
All quantities in the superblock, inodes, indirect blocks, etc. refer now
to this abstract unit (called "fsb" as it is in FFS) instead of disk sectors;
as a consequence segment summary blocks have to be multiples of a fragment in
size.  In v1 filesystems, compatibility code ensures that 1 fsb == 1 sector,
regardless of fragment size.

Fragments can now range in size between 512 and 32k; in the event that
LFS_LABELPAD (8k) is smaller than the disk address unit size, an extra
proto-superblock is kept at 8k from the beginning of the disk, to be used
*only* to locate the real superblocks.  (Not all of the userland knows about
this yet.)

Almost all of this was done not by me, but by joff.

Fixes, from joff, for correct compatibility operation of fsck_lfs, dumplfs,
lfs_cleanerd; correct usage message from dumplfs.

Import of what I've been calling "LFSv2", that is, LFS with some features
added that require changes to the on-disk data structures.  These include:

- 64-bit time in everything but inodes
- User-specified segment offset, and segment size no longer
  restricted to PO2.
- Serial number on segment summaries in addition to timestamp, and
  a new volume identifier, to make roll-forward feasible without
  fear of finding old data and thinking it was new.

Although I think this version works at least as well as what's on the trunk,
we're not done yet; hence this commit is going in on a branch and not on
the trunk.  Enhancements that are not here yet include fragment addressing,
like FFS does, instead of block addressing.

fix redundant decls

Fixed blockmap handling to properly use disk blocks rather than fragments.
Fixes an issue with fsck_lfs not detecting all duplicate blocks that may
exist in a corrupted filesystem.

use %ll_ instead of the less standard %q_

Sync w/ netbsd-1-5-base.

Add "-i" flag to specify the location of the index file inode, to
examine alternate checkpoints.  Regularize usage of maxino.  Remove olf
debugging cruft.

Convert to NetBSD source code style

fsck_lfs can now write to the filesystem, allowing it to correct most
(though still not all) errors in a damaged lfs.  Segment byte accounting
is corrected in pass 5.  "fsck_lfs -p" will do a partial roll-forward,
verifying the checkpoint from the newer superblock.  fscknames[] is
updated so that fsck knows about fsck_lfs.

Pull up revision 1.4 (requested by perseant):
  Fix name collision error due to recent kernel prototype updates.
  Fixes PR#9253.

Rename lfs_ifind so that it does not conflict with new kernel prototype.
Addresses PR #9253.

RCS Id police.

printf format fixes for Alpha.

Initial checkin of fsck_lfs.  This version cannot do any repair (-p flag
does nothing, and one of -p or -n is required) but can be useful as a
diagnostic tool.