CVS log for src/lib/librefuse/refuse.c

Up to [cvs.NetBSD.org] / src / lib / librefuse

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

lib/librefuse: Implement all sorts of compat tweaks to appease various file systems

ReFUSE now supports all the FUSE API variants from FUSE 1.1 to FUSE
3.10. Sorry for the freaking giant patch. I could not break it down
any further.

Zero-clear the fuse_context in fuse_destroy(3)

Support the FUSE option -ho

It is supposed to print a help message without the usage
line. Although it is deprecated and has been removed as of FUSE 3.0,
filesystems in the wild still use it.

Implement some missing functions that are part of the API

Correct the wrong prototype of fuse_daemonize(3) while retaining ABI compatibility

Change the way how FUSE_*_VERSION are handled

* FUSE_MAKE_VERSION(maj, min) now generates a 3-digits number if the
  version is higher than 3.9. This is needed to support FUSE 3.10 API.

* FUSE_{MAJOR,MINOR}_VERSION no longer have a fixed value but are
  derived from FUSE_USE_VERSION specified by the user code. This is
  needed to support more FUSE filesystems in the wild.

Implement a dummy pathconf() which always returns EINVAL

Do not call fuse_operations.getattr() before initializing filesystem

Increase the warning level to spot more mistakes

lib/librefuse: Add support for legacy types and functions

lib/librefuse: Implement FUSE session API and its signal handling functionality

librefuse: Preparation of a proper API versioning; no more #ifdef woes in user code

The goal is to fully support FUSE API version 3.0 while maintaining
API/ABI compatibility with code written for 2.6 (or even older).

* <fuse.h> now emits a compiler warning if it's included without
  defining FUSE_USE_VERSION. It had been silently defaulted to the
  latest supported version prior to this change. This is permissive
  compared to the original FUSE, as it emits an error instead.

* <fuse.h> now emits a warning if FUSE_USE_VERSION is higher than what
  can be provided.

* Added a macro FUSE_MAKE_VERSION(maj, min). It was missing from
  librefuse <fuse.h>.

No actual API updates have been made (yet).

Move the call of fuse_operations::init() from fuse_new() to fuse_loop()

Prior to this change we were calling init() before daemonizing the
process. Some filesystems call chdir(2) in init() but fuse_daemonize()
call chdir("/"), which breaks assumptions about the state of the
process.

Mostly merge changes from HEAD upto 20200411

Restore binary compatibility by using the statvfs90 structure internally.

Sync with HEAD

Use the equivalent FUSE_VERSION. Probably a more adequately named macro for this

provide fuse_version

Sync with HEAD.  (Note that most of these changes are simply $NetBSD$
tag issues.)

fuse_main(3): Support general fuse options via fuse_parse_cmdline(3)

* Prototypes for fuse_new(3), fuse_mount(3), and fuse_unmount(3) now
  matches to the original fuse interface.

* Add fuse_daemonize(3): needs to have a different prototype,
  otherwise we can't use puffs_daemon(3).

* Remove fuse_setup(3) and fuse_teardown(3). These obsolete functions
  has already been removed from the original interface.

* fuse_main(3) now supports the following command-line options
  compatible with the original fuse:

  -h, --help      print help message
  -V, --version   print library version (currently does nothing)
  -d, -o debug    enable debug output (PUFFS_FLAG_OPDUMP), implies -f
  -f              foreground mode
  -s              single threaded mode (always enabled for now)
  -o fsname=NAME  explicitly set the name of the file system

* fuse_main(3) now daemonizes the process by default. This is for the
  compatibility with the original fuse.

Change the way how puffs_fuse_node_create() behaves.

In puffs "create" and "open" are two separate operations with
atomicity achieved by locking the parent vnode. In fuse, on the other
hand, "create" is actually a create-and-open-atomically and the open
flags (O_RDWR, O_APPEND, ...) are passed via fi.flags. So the only way
to emulate the fuse semantics is to open the file with dummy flags and
then immediately close it.

You might think that we could simply use fuse->op.mknod all the time
but no, that's not possible because most file systems nowadays expect
op.mknod to be called only for non-regular files and many don't even
support it.

resync with head

sync with head

Pull up following revision(s) (requested by tron in ticket #1836):
	lib/librefuse/refuse.c: revision 1.96 via patch
FUSE seems to allow short writes without errors but PUFFS doesn't. Work
around this by returning ENOSPC in case of a short write to avoid protocol
errors. This change is based on problem analysis provided by Antti Kantee.
This fixes PR lib/45129 by myself.

Pull up following revision(s) (requested by tron in ticket #772):
	lib/librefuse/refuse.c: revision 1.96
FUSE seems to allow short writes without errors but PUFFS doesn't. Work
around this by returning ENOSPC in case of a short write to avoid protocol
errors. This change is based on problem analysis provided by Antti Kantee.
This fixes PR lib/45129 by myself.

FUSE seems to allow short writes without errors but PUFFS doesn't. Work
around this by returning ENOSPC in case of a short write to avoid protocol
errors. This change is based on problem analysis provided by Antti Kantee.

This fixes PR lib/45129 by myself.

sync with head

Set eofflag in puffs_fuse_node_readdir() from librefuse.

From Evgeniy Ivanov <lolkaantimat@gmail.com>

Pull up following revision(s) (requested by tron in ticket #1650):
	lib/librefuse/refuse.c: revision 1.94
Call the FUSE init operation before we try to access the file-system.
This prevents a crash in "fuse_ext2" which I previously worked around
with a patch.

Pull up following revision(s) (requested by tron in ticket #1648):
	lib/librefuse/refuse.c: revision 1.93
Don't ignore "userdata" argument in "fuse_main_real". This can crash
a FUSE file-system that passes a non-NULL argument here.

Call the FUSE init operation before we try to access the file-system.
This prevents a crash in "fuse_ext2" which I previously worked around
with a patch.

Don't ignore "userdata" argument in "fuse_main_real". This can crash
a FUSE file-system that passes a non-NULL argument here.

Sync with HEAD.

Third (and last) commit. See http://mail-index.netbsd.org/source-changes/2009/05/13/msg221222.html

fix typo (s/stucture/structure/)

sign-compare fix

fix -Wsign-compare issue

Sync with wrstuden-revivesa-base-2.

Add API version 2.6 variant of fuse_setup.

Reviewed by agc.

sync with HEAD

return -ret from statfs

sync with HEAD

Uncommit accidentally committed PUFFS_FLAG_OPDUMP.

overextensive library verbosity noticed by riz

puffs_fuse_node_create: if create() is specified set S_IFREG explicitly
to avoid passing unspecified mode. Fixes latest ntfs-3g... patch
from pooka tested by myself.

Rototill.

Ok, ok, a few more words about it: stop holding puffs_cc as a holy
value and passing it around to almost every possible place (popquiz:
which kernel variable does this remind you of?).  Instead, pass
the natural choice, puffs_usermount, and fetch puffs_cc via
puffs_cc_getcc() only in routines which actually need it.  This
not only simplifies code, but (thanks to the introduction of
puffs_cc_getcc()) enables constructs which weren't previously sanely
possible, say layering as a curious example.

There's still a little to do on this front, but this was the major
fs interface blast.

Remove "puffs_cid" from the puffs interface following l-removal
from the kernel vfs interfaces.  puffs_cc_getcaller(pcc) can be
used now should the same information be desired.

I've written more than a fair share of this to see my name up in
lights.  Wine & women, come to me now!

sync with HEAD

Pull the daemonizing code out of the library mainloop into the file
servers.  Calling daemon() (i.e. fork()ing) inside a library can
cause nice surprises for e.g. threaded programs.  As discussed with
Greg Oster & others.

Header cleanup: don't need defs.h to do a few includes for us.  Also,
use #include <> for global headers instead of #include "".

* Fix pthread key creation to depend on if it hasn't been created before
  instead of depending if a lock was succesfully acquired.  Also, add
  a comment explaining why it is such complex (for all I could guess).
* replace err()'s with abort()
* replace some abort()'s by returning failure

Use puffs prepost ops for setting pid/lwpid instead of separate
calls in all methods.

revert 1.77, MULTITHREADED_REFUSE has problems

Get rid of MULTITHREADED_REFUSE for context queries.  We can simply
use the pthread codepaths always.

Use new puffs features to provide correct caller info instead of
sometimes returning stale data.

Now that we don't dump operations by default, might as well detach
from the console by default also.  (but still would be nice if someone
provided options to toggle these using the fuse options framework)

In case a file system doesn't provide st_blksize, default to DEV_BSIZE.

Don't stay on the foreground and dump operations by default.

XXX: someone(tm) should make it possible to set PUFFS_OPFLAG_DUMP
using the fuse options framework

file refuse.c was added on branch matt-mips64 on 2007-07-18 22:05:42 +0000

Don't stay on the foreground and dump operations by default.

XXX: someone(tm) should make it possible to set PUFFS_OPFLAG_DUMP
using the fuse options framework

* add mntfromname parameter to puffs_init()
* set it in file servers

adapt: **newnode etc. pointers -> struct puffs_newinfo

adapt: pid -> const struct puffs_cid *

adapt: pcn->pcn_cred is now a pointer

Actually, keep PUFFS_KFLAG_NOCACHE and -o cache around as shorthand
to neither page- nor namecache.

PUFFS_KFLAG_NOCACHE became two, so introduce the command line options
-o nonamecache and -o nopagecache and adapt file systems where necessary.

Use pthread_[gs]etspecific to store a pointer to the fuse_context
information, enabled if MULTITHREADED_REFUSE is defined at build
time. This is not enabled by default since libpuffs is not
threadsafe yet.

librefuse is still not threadsafe, but this gets rid of a whole lot of
problems in the meantime.

Minor cleanup - change uses of malloc, memset to calloc.

Use EXIT_FAILURE in preference to the less informational 1.

Re-work the way the SET_FUSE_CONTEXT* macros are done, following
feedback from Antti.

# use puffs_cred_getuid(3) and puffs_cred_getgid(3)

for i in `jot 100 1`; do
	echo "I must respect the abstractions which pooka put in puffs_cred(3)"
done

Set the uid, gid and pid wherever possible (i.e. where known). This gets
us far enough along that fuse-loggedfs now works with the correct command,
uid and gid being displayed.

After we've malloc'd an area of storage, zero out this storage, "just in
case".

Calling puffs_exit() in fuse_{exit,unmount}() is wrong, as those
shouldn't release resources.  As a quick hack to prevent double
frees etc, introduce "dead" variable into fuse channel to signal
if the backing puffs resource was already freed.  XXX: the check
is only in exit/unmount currently.

Proper fix really depends on some puffs features I still haven't
gotten around to doing.

create hash for the root path object

initialize refusenode contents to 0

(don't change malloc to calloc and expect things not to break)

reflect changes in fs mount: use puffs_mount() instead of
puffs_domount() & puffs_start()

More fixes:
- make sure that the args array is NULL terminated.
- make sure argc is initialized and argv is set to NULL after freeing.
- make the deep copy function an opt interface function.

- Always check strdup return. For now we are lazy and bail instead of cleaning
  up.
- Use malloc and free instead of macros, delete macros.
- Merge the two copies of deep copy and free args that were slightly buggy
  into one and use that one.

XXX: Lots of ints should be size_t.
XXX: Bailing on error is not a good thing for a library.
XXX: Defs.h should be eliminated completely. Not all files need all the headers.

Enable puffs_fakecc.  FUSE file systems don't use continuations anyway,
and now file servers linked against libpthread won't crash when they
attempt to malloc something.

When parsing arguments for the directory to mount, search backwards from
the end of the argument vector for an argument that doesn't begin with '-',
and use it - this improves upon the previous naive approach of using the
last argument, which could have been placed there by one of the argument
appending functions.

Be consistent with the mount name processing, and always do the
puffs:refuse:filesystem substitution once and once only.

% priv mount.ntfs-3g ntfs-refuse.img /mnt
% df /mnt
Filesystem           1K-blocks      Used     Avail Capacity  Mounted on
puffs:refuse:ntfs-3g   1000943     66309    934634     6%    /mnt
% priv umount /mnt
% priv /usr/src/share/examples/refuse/id3fs/id3fs /mnt &
% df /mnt
Filesystem         1K-blocks      Used     Avail Capacity  Mounted on
puffs:refuse:id3fs         0         0         0   100%    /mnt
% priv umount /mnt
%

When puffs_mount fails, display the name of the directory which wasn't
able to be mounted.

Implement fuse_setup(), and fuse_teardown() - necessary for the python
bindings for refuse.

In case a fuse fs hands an stbuf with an inode number 0 to the
readdir filler, fake the inode number.  Our readdir() doesn't show
entries with inode number 0.

per discussion with Stephen Borrill

use PUFFS_FLAG_HASHPATH

Flesh out the args management a bit more:

+ put some meat into the add_arg and insert_arg functions
+ implement the free_args function
+ use 0 and 1 return values, instead of EXIT_SUCCESS and EXIT_FAILURE, in
  internal functions (they bear no relation to shell exit values)
+ deep copy argument structures in refuse.c - we need to do this because
  some FUSE file systems attempt to obliterate^Wcloak the argument vector
  "for security reasons"
+ a side benefit of this is that we get the specific refuse file system
  name back again:

[10:48:43] agc@inspiron1300 ...src/lib/librefuse 38 > df
Filesystem         1K-blocks      Used     Avail Capacity  Mounted on
/dev/dk0            28101396  19318530   7377798    72%    /
kernfs                     1         1         0   100%    /kern
procfs                     4         4         0   100%    /proc
ptyfs                      1         1         0   100%    /dev/pts
puffs:refuse:id3fs         0         0         0   100%    /mnt
[10:48:51] agc@inspiron1300 ...src/lib/librefuse 39 >

Instead of keeping on adding parameters to puffs_mount(), make it
only take the bare essentials, which currently means removing
"maxreqlen" from the argument list (all current callers I'm aware
of set it as 0 anyway).  Introduce puffs_init(), which provides a
context for setting various parameters and puffs_domount(), which
can be used to mount the file system.  Keep puffs_mount() as a
shortcut for the above two for simple file systems.

Bump development ABI version to 13.  After all, it's Friday the 13th.
Watch out!  Bad things can happen on Friday the 13th. --No carrier--

Hide struct puffs_usermount from the rest of the world and provide
accessors for interesting data in it.  Namely, you can now get
pu->pu_privdata with puffs_getspecific(), pu->pu_pn_root with
puffs_set/getroot() and pu->pu_maxreqlen with puffs_getmaxreqlen().

adapt to new readdir signature.  no functional change

use puffs_addvtype2mode() in puffs_fuse_node_mknod()

Revert the most recent changes I made - they weren't ready for primetime.

Fix a thinko - if  we're going to do a deep copy, let's do it properly.

With  thanks to xtraeme for the viirtual PR.

+ Only use EXIT_SUCCESS and EXIT_FAILURE for the return code to the
shell in the fuse_opt functions

+ add new debugging functions:
	+ __fuse_debug(), which is used to set and retrieve the
	debugging level, so that debugging information can be turned
	on and off from the user-level filesystem, and
	+ __fuse_pargs(), which prints out an argv vector

+ put the comment about pu_privdata back with the code it references

+ put the code to place the refuse name back into the fuse_new()
function - we have access to all the arguments there.

+ specifically include <fuse_opt.h> in <fuse.h> - this would seem to
be what FUSE itself does, judging by the number of fuse-based
filesystems I've patched to include <fuse_opt.h> specifically

+ do away with the forward reference to struct fuse_args in fuse.h,
since it is defined just after the forward reference without being
referenced in between

+ fill in the missing refuse_opt args functionality - add an arg to the
args structure in fuse_opt_add_arg() and fuse_opt_insert_arg(), and
free allocated memory in fuse_opt_free_args().

+ get rid of spurious forward declarations and prototypes.

+ if REFUSE_INHERIT_FS_CHARACTERISTICS is defined when refuse is built,
file systems inherit the vfs characteristics of their mountpoints. The
default value is NOT to inherit the vfs characteristics.

+ perform a deep copy of the fuse_args structure in fuse_mount() -
this is so that existing FUSE filesystems still DTRT when arguments
are zeroed out.

* Move fuse_opt* defs and prototypes into fuse_opt.h.
* Move fuse_opt* funcs from refuse.c into refuse_opt.c.

Implement fuse_opt_parse() and fuse_opt_match(). And make the other
functions just dummy, always returning 0 (I added debugging printfs
to see what the application is trying to do).

For now there are two things that do not work in fuse_opt:

	* options accepting arguments, i.e -otimeout=%u or -ofile=%s.
	* options without arguments are not enabled, just parsed.

At least now curlftpfs works, even with verbose mode! :-)

Ok'ed by pooka.

For just now, just use "puffs:refuse" as the file system name - this will
be revisited when thee fuse_args parsing is revisited.

Refactor a few interfaces a bit to look more like fuse.  fuse_args
is implemented as a very dummy version (i.e. totally unimplemented),
so some effort is still needed there.

After this change it is possible to compile and run ntfs-3g.  It
works read/write for ntfs images and shows no faults at least with
superficial testing.  I did not test it against a block device,
only an image.

Thanks to Tracy and Jason for help with the test image.

convert FREAD/FWRITE to O_RDWR-stuff in open

Implement readdir properly, or at least as close to possible to how
fuse does it: read directory completely into refuse buffers if starting
from read offset 0 and for later calls trickle results from the buffers
to the kernel without consulting the fuse file system.

revert 1.34, as discussed with agc.  It creates more problems than
it solves, namely breaks ~all fuse file systems.  Proper fix is known
but not yet implemented.

For large directories, remove the artificial restriction on one
buffer's worth of directory entries - this work is done in the higher
level (re)fuse readdir or getdir operation.

* missing open is not a failure, just return 0
* nuke accidentally committed debug printf

reverse order of arguments to fuse->op.symlink, remove comment
suspecting argument order reversal

* Thread flags to VOP_OPEN (except of course O_CREAT, O_TRUNC and O_EXCL)
  through fuse_file_info to the fuse open operation.
* reference count opens for each node and do release once the count
  drops to zero (instead of waiting for inactive).  I'm still not sure
  if this exactly matches linux/fuse behaviour, but at least we do
  open when a new file descriptor is opened for each node and a do
  a release when the reference to the last file descriptor for a
  node is closed, so it ought to be pretty close

cryptofs works after these changes

* create a refusenode also for the root node
* opendir directories in open

Implement a very very stubby fuse_unmount(), which does ... nothing.
Pending some puffs changes that are in the queue, this is the best way
currently.

fix return-in-the-middle bug in inactive.  and while there, make
it always return 0, since the return value is not used for error
indication for the kernel

puffs_fuse_node_inactive: initialize ret to 0 to avoid a warning.

* combine emulation layer tasks for node creation to a common routine
* make set/getattr easily callable from interface functions and sprinkle
  calls to locations which require them
* use libpuffs routine for stat->vattr translation instead of homegrown one
* I have concluded that we really want to do release/releasedir in
  inactive instead of reclaim.  make it so.  we'll tweak it for the next
  file system which decides it doesn't want to work

after these changes lzofs works, at least for most parts
(I'd assume the non-working parts to fail also on fuse/linux)

* don't allow reads beyond at offset = EOF, all fuse file systems can't
  handle it
* initialize refuse root node properly
* some misc cleanup

fix rename (yes, it's really that simple)

* oh, and when removing, remove the target node, not its parent directory
* use PUFFS_KFLAG_NOCACHE always for now, since code is too jumpy and
  volatile to deal with caches

* map reclaim to release, but call it only for files which were opened
* use node paths in remove/rmdir instead of pcn full paths (which are
  null for those ops)

in create, if op.create doesn't exist, try op.mknod instead

Added extra functionality:

	fuse_opt_free_args()

	fuse_exit()

Both from Juan RP - thanks!

support some more fuse interfaces required for setattr:
utimens, utime, truncate and ftruncate

* can't call unlink(2) & rmdir(2) in error paths.  Rather, call
  ops specified in the fuse op vector (or, if they're missing,
  we're SOL)
* great return value overhaul: return -ret to get linuxy -errno
  right for the kernel

write to end-of-file in case of IOAPPEND

* first stab at create
* return proper return values from read/write

No need to set read, readdir and write in the ops vector twice.
Pair up some ops while shuffling.

Readlink should place the length of the link (without terminating
nul) in the length field.  Make it so.

* slightly better dirfillers
* make readlink work

implement proper lookup

Implement a very very hacky fuse_get_context(), needs to be visited once
threading is added (and probably otherwise also).

link operation does not create a new node, therefore don't do it in
the code either

need to compare against PUFFS_VNOVAL instead of 0 to determnine if
fields are valid

pass real fuse_file_info to rest of fuse ops instead of a memset-to-zero one

sprinkle some KNF

agc ok

fuse directory fillers return opposite truth values from puffs_nextdent,
so invert return value to make readdir callers which check the return
value work.

Allocate refuse private data, refusenode, for all nodes.  Currently it
carries fuse_file_info.

.. and lo, cddafs works

implement getdir, used by some older fuse file systems instead of readdir

Don't release nodes in _remove or _rmdir, implement reclaim
where to do it.

* sprinkle some comments to obvious XXX places
* remove outdated #ifdef from write
* return 0 if unmount is not supported instead of ENOSYS

Add an implementation of the file system in userspace functionality,
based on top of libpuffs.  This version is still barebones and
incomplete, but will benefit from others working on it, rather than
just me.

The option code has still to be implemented, but this gets far enough
to run hellofs (from the fuse web page):

	% priv ./hellofs ~/hellofs/mnt &
	% l ~/hellofs/mnt
	total 2
	drwxr-xr-x  2 root  wheel    0 Jan  1  1970 .
	drwxr-xr-x  4 agc   agc    512 Feb  9 18:05 ..
	-r--r--r--  1 root  wheel   13 Jan  1  1970 hello
	% cat ~/hellofs/mnt/hello
	Hello World!
	% df ~/hellofs/mnt
	Filesystem           1K-blocks      Used     Avail Capacity  Mounted on
	/dev/dk0              28101396  18946682   7749646    70%    /
	kernfs                       1         1         0   100%    /kern
	procfs                       4         4         0   100%    /proc
	ptyfs                        1         1         0   100%    /dev/pts
	puffs:refuse:hellofs  28101396  18946682   7749646    70%    /home/agc/hellofs/mnt
	% mount -v -v
	...
	mount: mount_puffs not found for /home/agc/hellofs/mnt
	puffs:refuse:hellofs on /home/agc/hellofs/mnt type puffs (nosuid, nodev, fsid: 0xcb01/0x6acb, reads: sync 0 async 0, writes: sync 0 async 0)
	%

This code is not enabled by default. Yet.