CVS log for src/lib/libpuffs/puffs.c

Up to [cvs.NetBSD.org] / src / lib / libpuffs

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

lib: remove CONSTCOND comment

Since 2021-01-31, lint doesn't need it anymore for the common pattern of
'do ... while (0)'.

Allow calling puffs_mount(3) before puffs_daemon(3)

puffs_daemon(3) creates a pipe before forking, and the parent process
waits for the child to either complete puffs_mount(3) or fail. If a
user calls puffs_daemon(3) after puffs_mount(3), the function
deadlocks. While this error-reporting functionality is really a nice
thing to have, deadlocking is not great. If the filesystem has already
been mounted, puffs_mount(3) should just daemonize the process and
return.

This became an issue because fuse_daemonize(3) in FUSE API had no such
requirement and some FUSE filesystems in the wild suffered deadlocks.

Use size_t in place of voff_t in puffs_setrootinfo(3)

Chuck Silvers pointed out that voff_t was also supposed to be
kernel-only. The correct type to use in userland would be off_t, but
since changing vsize_t to either voff_t or off_t is an ABI change on
32-bit platforms, we use size_t knowing that it is technically
incorrect.

Avoid using register_t or vsize_t in <puffs.h>

<puffs.h> is a user-space header, and should not use any of
kernel-only types. It's not reasonable to require user-land
filesystems to #define _KERNTYPES.

puffs(3): Replace realloc(x * y) with reallocarr

Sync with HEAD

Sync with HEAD

use a switch.

Typos.

fix error messages (use __func__, quotes)
also differentiate between canonical and non-relative.

Adjust to prototype.

fix error messages

Do not warn about relative path because of trailing slash

libpuffs calls realpath() to obtain an absolute path to use for mounting.
If the obtained path is different from the one given by the caller, a
warning is issued. This included the situation where the path passed by
the caller just have trailing slashes, a situation where we just want them
to be striped without a warning.

Pull up following revision(s) (requested by manu in ticket #181):
	lib/libperfuse/fuse.h: revision 1.6
	lib/libperfuse/ops.c: revision 1.78
	lib/libperfuse/perfuse.c: revision 1.35
	lib/libperfuse/perfuse_priv.h: revision 1.36
	lib/libpuffs/dispatcher.c: revision 1.48
	lib/libpuffs/opdump.c: revision 1.37
	lib/libpuffs/puffs.c: revision 1.118
	lib/libpuffs/puffs.h: revision 1.126
	lib/libpuffs/puffs_ops.3: revisions 1.40-1.41
	sys/fs/puffs/puffs_msgif.h: revision 1.82-1.83
	sys/fs/puffs/puffs_msgif.h: revision 1.82
	sys/fs/puffs/puffs_vnops.c: revision 1.196
Add PUFFS support for fallocate and fdiscard operations
--
libpuffs support for fallocate and fdiscard operations
--
Add PUFFS_HAVE_FALLOCATE in puffs_msgif.h so that filesystem can decide
at build time wether fallocate is usable
--
FUSE fallocate support
There seems to be no fdiscard FUSE operation at the moment, hence that
one is left unused.

libpuffs support for fallocate and fdiscard operations

sync with head

Pull up following revision(s) (requested by manu in ticket #1706):
	lib/libpuffs/puffs.c: revision 1.108
signal pathconf in the vector of supported ops

fix crashes caused by using the results of getcontext()
after the caller returns.

Pull up following revision(s) (requested by manu in ticket #1645):
lib/libc/sys/Makefile.inc                       1.207 via patch
lib/libc/sys/extattr_get_file.2                 patch
lib/libpuffs/dispatcher.c                       1.34,1.36 via patch
lib/libpuffs/puffs.c                            1.107 via patch
lib/libpuffs/puffs.h                            1.115,1.118 via patch
sys/fs/puffs/puffs_msgif.h                      1.71,1.76 via patch
sys/fs/puffs/puffs_vfsops.c                     1.88 via patch
sys/fs/puffs/puffs_vnops.c                      1.145,1.154 via patch
sys/kern/vfs_xattr.c                            1.24-1.27 via patch
sys/kern/vnode_if.c                             1.87 via patch
sys/sys/Makefile                                1.133 via patch
sys/sys/extattr.h                               1.6 via patch
sys/sys/vnode_if.h                              1.81 via patch
sys/ufs/ffs/ffs_vnops.c                         patch
sys/ufs/ufs/ufs_extattr.c                       1.31,1.34 via patch

* support extended attributes
* bump major due to structure growth
* add some spare space
* remove ABI sillyness
Support extended attributes.
Fix multiple non compliances in our Linux-like extattr API, and make it
public so that it can be used.
Improve a bit listxattr(2). It attemps to list both system and user
extended attributes, and it faled if calling user did not have privilege
for reading system EA. Now we just lise user EA and skip system EA in
reading them is not allowed.
Fix bug introduced in previous commuit: Do not vrele() a vnode we did not
obtained.
Improve UFS1 extended attributes usability
- autocreate attribute backing file for new attributes
- autoload attributes when issuing extattrctl start
- when autoloading attributes, do not display garbage warning when looking
up entries that got ENOENT
Add a flag to VOP_LISTEXTATTR(9) so that the vnode interface can tell the
filesystem in which format extended attribute shall be listed.
There are currently two formats:
- NUL-terminated strings, used for listxattr(2), this is the default.
- one byte length-pprefixed, non NUL-terminated strings, used for
  extattr_list_file(2), which is obtanined by setting the
  EXTATTR_LIST_PREFIXLEN flag to VOP_LISTEXTATTR(9)
This approach avoid the need for converting the list back and forth, except
in libperfuse, since FUSE uses NUL-terminated strings, and the kernel may
have requested EXTATTR_LIST_PREFIXLEN.

Pull up following revision(s) (requested by manu in ticket #1623):
	lib/libpuffs/puffs.c: revision 1.116 via patch
	sys/fs/puffs/puffs_vnops.c: revision 1.151 via patch
Call advlock method if supplied

Call advlock method if supplied

Remove PUFFS_WITH_THREADS ifdef garbage, just use libc weak symbols.

unwhine lint

stat mount target path only if we're actually goint to use it.
This avoids errors from the comfd case when the mount target does
not exist in our local namespace.

set time_t compat where necessary

When using the top secret PUFFS_COMFD model, write out size of the
puffs kernel args before the contents.  This allows parties which
have no clue about the contents to proxy the protocol easily.

Reenable pathconf now that the retval argument is register_t.

Actually, disable pathconf for a while longer: the retval needs a
little finetuning (not that everyone is jumping right to using it,
but fixing will have to wait until tomorrow, and who knows what
i'll forget during the night and how long it will eventually take).

signal pathconf in the vector of supported ops

* support extended attributes
* bump major due to structure growth
* add some spare space
* remove ABI sillyness

remove three years out-of-date comment

sync to netbsd-5

Add convenience routine puffs_unmountonsignal(), which does exactly that.

Use typecast instead of ULL for a less whiny lint on LP64.

Rewrite puffs_exit() in terms of the PUFFSOP_UNMOUNT kernel request.
Now server exit is a lot quite cleaner than previously.

Set high bit on server->kernel requests to better differentiate
between the direction of the request.

Add an error handler which syslogs but does not abort() and make it
the default.  Most servers just shovel bits from their backend to
the kernel without checking the contents and dumping core in case
the backend gives garbage is a bit harsh.

Pull up following revision(s) (requested by pooka in ticket #1105):
	lib/libpuffs/puffs.h: revision 1.112
	lib/libpuffs/puffs.c: revision 1.100
	distrib/sets/lists/base/md.amd64: patch
	distrib/sets/lists/base/md.sparc64: patch
	distrib/sets/lists/base/shl.mi: patch
	lib/libpuffs/shlib_version: patch

Make it possible to supply parameters to puffs_init() later (but
before puffs_mount()).

Make it possible to supply parameters to puffs_init() later (but
before puffs_mount()).

Pull up the following revisions(s) (requested by pooka in ticket #1100):
	lib/libpuffs/dispatcher.c:	revision 1.33
	lib/libpuffs/puffs.c:	revision 1.99
	lib/libpuffs/puffs.h:	revision 1.111
	sys/fs/puffs/puffs_msgif.h:	revision 1.67 via patch
	sys/fs/puffs/puffs_vnops.c:	revision 1.136

Support VOP_ABORTOP() in puffs.

Dispatch puffs_node_abortop().

Note: We use the storage of puffs_cache_write from puffs_ops for
this purpose.  It's not issued by the kernel and hence currently
unused, and this saves us from the trouble of bumping the lib major
version.

Pull up the following revisions(s) (requested by pooka in ticket #1095):
	lib/libpuffs/puffs.c:	revision 1.94
	lib/libpuffs/puffs.h:	revision 1.110

Add puffs_cancel(), which allows to back down after puffs_init()
and before puffs_mount().

Pull up the following revisions(s) (requested by pooka in ticket #1094):
	lib/libpuffs/puffs.3:	revision 1.45
	lib/libpuffs/puffs.c:	revision 1.93
	lib/libpuffs/puffs.h:	revision 1.109

Add function puffs_setspecific(pu, private) which can be used to set the
specific data after the call to puffs_init().

fix some sign comparison warnings

Actually we need 'long int' to avoif a stupid 'truncating long to int'
warning.

Change type of 'n' to 'int' so that it can be printed with %d.
The domain of the value is [-1..sizeof (int)] so a type of 'int' is fine.

Close write-end of daemonpipe in reader to get EOF if the writer
goes awol.

Add puffs_cancel(), which allows to back down after puffs_init()
and before puffs_mount().
(this should actually be integrated with puffs_exit(), but that
routine is currently so non-sensible that I didn't bother now)

add:
     puffs_setspecific(pu, private)
	Can be used to set the specific data after the call to
	puffs_init().

Sync with wrstuden-revivesa-base-2.

Replace void * by puffs_cookie_t where appropriate.  No functional change.

clear cached puffs_cc's in puffs_exit()

* make sure we come out of mainloop in the main context.  otherwise in
  certain cases the server would jump to hyperspace when unmounted
* sprinkle some debug prints

sync with HEAD

Implement lazy contexts for file systems using puffs_mainloop().
Previously each request was executed on its own callcontext and
switched to every time the request was being processed.  Now requests
borrow the mainloop context and switch only if/when they yield.
This takes two context switches away from a file system request
bringing down the typical amounts 2->0 (e.g. dtfs) and 4->2 (e.g.
psshfs).

The interfaces for manually executing requests changed a bit:
puffs_dispatch_create() and puffs_dispatch_exec() must now be used.
They are not tested, as nothing in-tree wants them and I doubt
anyone else is really interested in them either.

Also do some misc code cleanup related to execution contexts.  The
"work-in-progress checkpoint" committed over a year ago was starting
to look slightly weed-infested.

Cache a number of execution contexts instead of re-initializing
them every time.  Speeds up pure in-memory file systems such as
sysctlfs or dtfs quite a bit.  For actual I/O-workhorses the result
is of course less tasty.

Use PROT_NONE page to protect bottom of stack from overrun

Introduce PUFFS_STACKSIZE_MIN to avoid overloading 0 and getting
a warning from the library when really wanting to say "just the
minimum, please".

sync with HEAD

liberate gas-huffin crack-puffin' assert

fix pasto

Also pass fromname in the "distributed" case.

Provide some transparent "support" for "distributed" file systems.
If the env variable PUFFS_COMFD is set, the descriptor value
contained in it is used for communication instead of opening
/dev/puffs and doing mount(2).

This feature is obviously very undocumented and should not be used
without adult supervision.

Add a public interface for setting a cookie mapper.

Send a response message for flush operations from the kernel instead
of abusing the return value of write(2).

Part 3/n of extensive changes to message handling:

Get rid of the original puffs_req(3) framework and use puffs_framebuf(3)
instead for file system requests.  It has the advantage of being
suitable for transporting a distributed message passing protocol
and therefore us being able to run the file system server on any
host.

Ok, puffs is not quite here yet: libpuffs needs to grow request
routing support and the message contents need to be munged into a
host independent format.  Saying which format would be telling,
but it might begin with an X, end in an L and have the 13th character
in the middle.  Keep an eye out for the sequels: Parts 3+m/n.

* allocate pcc as a slice from the stack instead of allocating it
  separately
* provide puffs_cc_getcc()

This is in preparation for the removal of you-should-guess-what as
an argument to routines here and there and everywhere.

print more info when receiving an abort

puffs_mount():
* initialize values better
* always go through "fail" to see if puffs_daemon() needs to be
  woken up
* open device r/w instead of r/o

Introduce puffs_daemon(), which works like daemon(3) except that it
also synchronizes with puffs_mount() and does not return (exit) in the
parent process until the file system has been mounted.  This makes
it possible to reliably run e.g. mount_foo jippi /kai ; cd /kai/ee

sync with -HEAD

sync with HEAD

Open the kernel descriptor as part of mount(), not init().  Then
it doesn't matter if someone fork()s or does other tricks between
init() and mount() (and besides, now it's where it logically should be).

Pull the daemonizing code out of the library mainloop into the file
servers.  Calling daemon() (i.e. fork()ing) inside a library can
cause nice surprises for e.g. threaded programs.  As discussed with
Greg Oster & others.

Sprinkle some hackish locks here and there, just enough to allow
libp2k & rump to run multithreaded.

Add possibility to set pre- and post callbacks which will be called
for all operations before or after executing the specific callback
(the api is not be final yet, though).

Add some very initial support for a threading worker model as an
alternative to the (vastly superior ;) continuation model.  This
is very preliminary stuff and not compiled by default (which it
even won't do without some other patches I cannot commit yet).

The raison d'commit of the patch is a snippet which ensures proper
in-order dispatching of all operations, including those which don't
require a response.  Previously many of them would be dispatched
simultaneosly, e.g. fsync and reclaim on the same node, which
obviously isn't all that nice for correct operation.

mmap stacks, makes it possible to align them

Make pu_haskq a flag instead of its own variable.

Add puffs_cc_schedule() which marks a pcc runnable and will pass
execution to it when in the main loop the next time.

Part 1/n of some pretty extensive changes to how the kernel module
interacts with the userspace file server:

  * since the kernel-user communication is not purely request-response
    anymore (hasn't been since 2006), try to rename some "request" to
    "message".  more similar mangling will take place in the future.

  * completely rework how messages are allocated.  previously most of
    them were borrowed from the stack (originally *all* of them),
    but now always allocate dynamically.  this makes the structure
    of the code much cleaner.  also makes it possible to fix a
    locking order violation.  it enables plenty of future enhancements.

  * start generalizing the transport interface to be independent of puffs

  * move transport interface to read/write instead of ioctl.  the
    old one had legacy design problems, and besides, ioctl's suck.
    implement a very generic version for now; this will be
    worked on later hopefully some day reaching "highly optimized".

  * implement libpuffs support behind existing library request
    interfaces.  this will change eventually (I hate those interfaces)

Pass reason string to pu_errnotify.  Print it in the default
version before aborting.

silence lint.  also noticed by xtraeme

Add error notifications, which are used to deliver errors from the
kernel to the file server for silly things the file server did,
e.g. attempting to create a file with size VSIZENOTSET.  The file
server can handle these as it chooses, but the default action is
for it to throw its hands in the air and sing "goodbye, cruel world,
it's over, walk on by".

run realpath() on the mountpath

Add puffs_framev_enqueue_waitevent, which waits for read/write/error
events from an i/o descriptor.  It yields until an event happens
and continues execution when the event loop notices the event has
happened.

file puffs.c was added on branch matt-mips64 on 2007-07-20 13:14:56 +0000

Add puffs_framev_enqueue_waitevent, which waits for read/write/error
events from an i/o descriptor.  It yields until an event happens
and continues execution when the event loop notices the event has
happened.

Don't float puffs_kargs needlessly around after mount, saves
a couple k of memory (and there was much rejoycing!)

Use ioctl(PUFFSREQSIZEOP) instead of mount(MNT_GETARGS) to fetch
the maximum request size after mount.  Calling mount(MNT_GETARGS)
from the file server is currently not kosher, as it vrele()s the
root vnode, potentially causing an inactive, which the file server
cannot handle while it itself is inactive in the kernel (deadlock).

* add mntfromname parameter to puffs_init()
* set it in file servers

Add new 'data_len' parameter to mount system call.
Add an additional call with MNT_GETARGS to retrieve the modified
information instead of relying on the kernel code abusing the mount
interface during mount.

Make puffs_framebuf better suited from environments with multiple sources:

* in addition add/remove, allow enable/disable, which can be used
  to control events for descriptors without having to remove all the
  data associated with them
* add directsend/receive, which can be used to pass the same buffer
  from the caller to read/writeframe and back again
* add flags to enqueue functions and allow urgent buffers to be
  processed as the next PDU

* reorder calls to get results processed even if our I/O descriptors
  go back to zero
* don't give up on the read-side of an fd if there is still unreceived data

Get rid of the "int *refs" parameter to inactive: the same can be
accomplished now with puffs_setbacks.

print a specific warning if /dev/puffs cannot be opened in init

Move puffs to a two clause license where it already isn't so.  And
as agc pointed out, even files with the third clause were already
effectively two clause because of a slight bug in the language...

remember to put the results of the final operations (including unmount)
to the kernel in mainloop

* stuff results back to kernel before calling kevent - the file server
  loop function might generate some results.  and this is still "after"
  event handling (except for the first call, but I'm not too keen on
  optimizing for that)
* don't be such a baby about EINTR from kevent().  if we get it, suck
  it up and continue instead of quitting

Reflect kernel changes for supplying root node information in mount
instead of puffs_start().  Get completely rid of puffs_start(), as
everything it used to do is now handled by the mount routine.
Introduce an optional pre-mount call puffs_setrootinfo() for setting
non-default root node information.  As the old puffs_mount() is
now virtually useless, say byebye to it and rename the old
puffs_domount() to puffs_mount(), but add a root cookie parameter
to compensate for the late puffs_start().

switch the arbitrary default errno value for network i/o from
ECONNABORTED to ECONNRESET, which is at least a little bit better

memset freshly allocated puffs_usermount to 0

Step 3 and currently final step of a more general purpose event loop:
support removal and addition of i/o file descriptors on the fly.

* detect closed file descriptors
* automatically free waiters of a dead file descriptor
* give the file server the possibility to specify a callback which
  notifies of a dead file descriptor
* move loop function to be a property of the mainloop instead of
  framebuf (doesn't change effective behaviour)
* add the possibility to configure a timespec parameter which
  attempts to call the loop function periodically
* move the event loop functions from the puffs_framebuf namespace
  to puffs_framev to differential between pure memory management
  functions

move puffs request dispatching code into it's own module.
no functional change

Step 2 of general-purpose eventloop: get rid of puffs_framebuf_eventloop().
puffs_mainloop() now handles that functionality as well.

Add some kludges to be able to execute requests on the regular program
stack instead of the continuation stack.  This is for lib/36011, where
pthread gets confused since we aren't running on the regular stack.
I'm not really sure which direction to go to with this quite yet, so
make the hack hard to enable on purpose.  The whole request dispatch
code needs cleaning anyway.

Remove the puffs_usermount argument from puffs_req_handle().  It can
be found from the other arguments and this way puffs_req_handle() is
more symmetric with the other puffs_req() routines.

Return 0 from puffs_mainloop() if no error occurs.
pointed out by Phil Nelson

Add puffs_setback(), which can be used to set setbacks for nodes before
the call returns to the kernel.  Currently it can be called only from
open(), mmap(), remove() and rmdir().  (But that might change)

assert that the file server doesn't try to return more cookies than
it was supposed to

alloc to set flags when setting fhsize

some routines for setting mount parameters

Instead of keeping on adding parameters to puffs_mount(), make it
only take the bare essentials, which currently means removing
"maxreqlen" from the argument list (all current callers I'm aware
of set it as 0 anyway).  Introduce puffs_init(), which provides a
context for setting various parameters and puffs_domount(), which
can be used to mount the file system.  Keep puffs_mount() as a
shortcut for the above two for simple file systems.

Bump development ABI version to 13.  After all, it's Friday the 13th.
Watch out!  Bad things can happen on Friday the 13th. --No carrier--

Hide struct puffs_usermount from the rest of the world and provide
accessors for interesting data in it.  Namely, you can now get
pu->pu_privdata with puffs_getspecific(), pu->pu_pn_root with
puffs_set/getroot() and pu->pu_maxreqlen with puffs_getmaxreqlen().

* support nodetofh and fhtonode callbacks
* use new signature of node_readdir

initial support for cacheops

Until someone can sort non-priviledged mounts and kauth, force
MNT_NOSUID | MNT_NODEV for mounts with geteuid() != 0.  I'm tired
of typing them on the command line every time I test a file system.

dump return values for operations

Introduce puffs_path_walkcmp(), which is to be called from nodewalk
and compares the path of the node against the given pathobject.
Also make comparison method take a flag to indicate if it should
check if the second path is a true prefix of the first.

plus some namespace cleanup

* get rid of the holy kernel-given ISDOTDOT, and instead define a
  macro which does strcmp against ".." and (the untranslated)
  componentname
* make PUFFS_FLAG_BUILDPATH build paths also if dotdot is the case,
  and adapt the regular path objects to this
* make nullfs lookup readable because we can now get rid of dotdot
  processing there

add interfaces for suspending the file system

Be consistent with the rest of the framework about the argument
order of puffs_docc and puffs_dopreq

be consistent in naming request handlers

revoke revoke

Checkpoint some more work in progress: for the benefit those file
system backends which operate purely based on paths, push out more
path management into the library and make path management more
abstract: enable a file system to define a bunch of path management
callbacks, which are used by the framework.  Management of normal
/this/is/a/path type paths is provided by the library.

For directory renames we need to rename all the child nodes for
the given directory if the file system wants paths (PUFFS_FLAG_BUILDPATH).
Do this by walking the nodelist and adjusting the path prefix of
each matching node.

* don't chdir in possible daemon() call for the benefit of file systems
  which specify a relative path as the root
* if (buildpath), build link source full pathname for the duration of
  the operation

* get rid of the mount callback; it's no great surprise to the
  server that it needs to mount the file system backend if it wants
  to call mount
* provide some options for getmntopts(), assume that callers will parse
  command line (or fstab) args
* reorganize the puffs_cc interface just a bit, preparing for a bigger
  revamp later

* inform kernel of our version
* homegrown lib ABI check (to avoid bumping major a ludicrillion times
  before reaching stability)

checkpoint some experimental work-in-progress, namely:

Add support for having multiple outstanding operations.  This is done
by exposing enough interfaces so that it is convenient to have the
main event loop in the implementation itself and by providing a
continuation framework for convinient blocking and rescheduling.

works fine, but will undergo further cleanup & development

CID4360: don't leak `buf' upon return
ok pooka@

outline & export code for interfacing with kernel GET/PUT requests

g/c out-of-date comment

support mmap

cleanup generated stuff from comments

rototill the operation callbacks: unify the ops under struct puffs_ops
and namespace them

adapt to kernel change:
Allow multiple requests to be transferred in each GET/PUTOP

PUFFSFLAG -> PUFFS_FLAG to be consistent with the kernel

build & send implemented operations list to kernel

summon daemon(3) in mainloop unless the nodaemon flag is given

* call statvfs() at mount() time, pass that info along
* start is gone

shuffle flags a bit

* catch up with kernel changes
* better error handling when mounting

* support FAF (Fire-And-Forget) class operations
* adapt to new lookup

pass rdev for specfs support

handle PUFFS_VN_INACTIVE

pass lint; fix install.

add libpuffs - the userspace programming interface for puffs

This work was initially started and completed for Google SoC 2005
and tweaked to work a bit better in the past few weeks.  While
being far from complete, it is functional enough to be able and
stable to host a fairly general-purpose in-memory file system in
userspace.  Even so, puffs should be considered experimental and
no binary compatibility for interfaces or crash-freedom or zero
security implications should be relied upon just yet.

The GSoC project was mentored by William Studenmund and the final
review for the code was done by Christos.