![[BACK]](/icons/back.gif){kind=icon}
Up to [cvs.NetBSD.org] / src / lib / libpuffs
Request diff between arbitrary revisions
Keyword substitution: kv
Default branch: MAIN
Use size_t in place of voff_t in puffs_setrootinfo(3) Chuck Silvers pointed out that voff_t was also supposed to be kernel-only. The correct type to use in userland would be off_t, but since changing vsize_t to either voff_t or off_t is an ABI change on 32-bit platforms, we use size_t knowing that it is technically incorrect.
Avoid using register_t or vsize_t in <puffs.h> <puffs.h> is a user-space header, and should not use any of kernel-only types. It's not reasonable to require user-land filesystems to #define _KERNTYPES.
Mostly merge changes from HEAD upto 20200411
Restore binary compatibility by using the statvfs90 structure internally.
consistency fixes.
Pull up following revision(s) (requested by manu in ticket #181): lib/libperfuse/fuse.h: revision 1.6 lib/libperfuse/ops.c: revision 1.78 lib/libperfuse/perfuse.c: revision 1.35 lib/libperfuse/perfuse_priv.h: revision 1.36 lib/libpuffs/dispatcher.c: revision 1.48 lib/libpuffs/opdump.c: revision 1.37 lib/libpuffs/puffs.c: revision 1.118 lib/libpuffs/puffs.h: revision 1.126 lib/libpuffs/puffs_ops.3: revisions 1.40-1.41 sys/fs/puffs/puffs_msgif.h: revision 1.82-1.83 sys/fs/puffs/puffs_msgif.h: revision 1.82 sys/fs/puffs/puffs_vnops.c: revision 1.196 Add PUFFS support for fallocate and fdiscard operations -- libpuffs support for fallocate and fdiscard operations -- Add PUFFS_HAVE_FALLOCATE in puffs_msgif.h so that filesystem can decide at build time wether fallocate is usable -- FUSE fallocate support There seems to be no fdiscard FUSE operation at the moment, hence that one is left unused.
Pull up following revision(s) (requested by manu in ticket #1140): lib/libperfuse/ops.c 1.63-1.69 lib/libperfuse/perfuse.c 1.32-1.33 lib/libperfuse/perfuse_priv.h 1.32-1.34 lib/libperfuse/subr.c 1.20 lib/libpuffs/creds.c 1.16 lib/libpuffs/dispatcher.c 1.47 lib/libpuffs/puffs.h 1.125 lib/libpuffs/puffs_ops.3 1.37-1.38 lib/libpuffs/requests.c 1.24 sys/fs/puffs/puffs_msgif.h 1.81 sys/fs/puffs/puffs_sys.h 1.85 sys/fs/puffs/puffs_vnops.c 1.183 usr.sbin/perfused/msg.c 1.22 Bring libpuffs, libperfuse and perfused on par with -current: - implement FUSE direct I/O - remove useless code and warnings - fix missing GETATTR bugs - fix exended attribute get and list operations
libpuffs support for fallocate and fdiscard operations
Pull up following revision(s) (requested by manu in ticket #52): lib/libperfuse/perfuse_priv.h: revision 1.33 lib/libperfuse/ops.c: revision 1.67 lib/libpuffs/dispatcher.c: revision 1.47 lib/libperfuse/perfuse_priv.h: revision 1.34 lib/libperfuse/ops.c: revision 1.68 lib/libperfuse/ops.c: revision 1.69 lib/libpuffs/puffs_ops.3: revision 1.37 lib/libpuffs/puffs_ops.3: revision 1.38 lib/libperfuse/perfuse.c: revision 1.32 lib/libperfuse/perfuse.c: revision 1.33 usr.sbin/perfused/msg.c: revision 1.22 lib/libpuffs/puffs.h: revision 1.125 Implement FUSE direct I/O. Remove useless code and warnings
Add an open2 method, like open but with an additionnal oflags used by the filesystem to send back information about the file. This is used to implement PUFFS_OPEN_IO_DIRECT by which the filesystem tells the kernel that read/write should bypass the page cache.
sync with netbsd-6-0-RELEASE.
sync with head
Pull up following revision(s) (requested by manu in ticket #525): lib/libpuffs/dispatcher.c: revision 1.44 lib/libp2k/p2k.c: revision 1.57 lib/libpuffs/puffs_ops.3: revision 1.35 lib/libpuffs/pnode.c: revision 1.13 lib/libpuffs/puffs.3: revision 1.56 lib/libpuffs/puffs.h: revision 1.124 Fix regression that has been introduced when the lookup/reclaim race condition was addressed in libpuffs by counting lookups. The fix assumes that cookies map to struct puffs_cookie, which has not been documented as a requirement for filesystems using libpuffs. As an example, we got burnt by this assumption in libp2k (kern/46734), and we fixed bit by actually mapping libp2k cookies to struct puffs_node. It is unlikely, but there may be third party filesystems that use cookies unmapped to struct puffs_node, and they were left broken for now. - we introduce a puffs_init() flag PUFFS_FLAG_PNCOOKIE that let filesystems inform libpuffs that they map cookies to struct puffs_node. Is that flag is used, the lookup/reclaim race condition fix is enabled. We enable the flag for libp2k. - filesystems that use puffs_pn_new() obviouslty use struct puffs_node and gain PUFFS_FLAG_PNCOOKIE automatically even if they did not specify it in puffs_init(). This include all our PUFFS filesystem in-tree except libp2k. - for filesystems not willing to use struct puffs_node, we introduce a reclaim2 vnop, which is reclaim with an additionnal lookup count argument. This vnop let the filesystem implement the lookup/reclaim race fix on its own.
Fix regression that has been introduced when the lookup/reclaim race condition was addressed in libpuffs by counting lookups. The fix assumes that cookies map to struct puffs_cookie, which has not been documented as a requirement for filesystems using libpuffs. As an example, we got burnt by this assumption in libp2k (kern/46734), and we fixed bit by actually mapping libp2k cookies to struct puffs_node. It is unlikely, but there may be third party filesystems that use cookies unmapped to struct puffs_node, and they were left broken for now. - we introduce a puffs_init() flag PUFFS_FLAG_PNCOOKIE that let filesystems inform libpuffs that they map cookies to struct puffs_node. Is that flag is used, the lookup/reclaim race condition fix is enabled. We enable the flag for libp2k. - filesystems that use puffs_pn_new() obviouslty use struct puffs_node and gain PUFFS_FLAG_PNCOOKIE automatically even if they did not specify it in puffs_init(). This include all our PUFFS filesystem in-tree except libp2k. - for filesystems not willing to use struct puffs_node, we introduce a reclaim2 vnop, which is reclaim with an additionnal lookup count argument. This vnop let the filesystem implement the lookup/reclaim race fix on its own.
Pull up following revision(s) (requested by manu in ticket #438): lib/libperfuse/perfuse_priv.h: revision 1.31 sys/fs/puffs/puffs_msgif.h: revision 1.80 sys/fs/puffs/puffs_vnops.c: revision 1.171 lib/libpuffs/puffs_ops.3: revision 1.31 sys/fs/puffs/puffs_vnops.c: revision 1.172 sys/fs/puffs/puffs_vnops.c: revision 1.173 sys/fs/puffs/puffs_vnops.c: revision 1.174 usr.sbin/perfused/perfused.c: revision 1.24 sys/fs/puffs/puffs_sys.h: revision 1.80 sys/fs/puffs/puffs_sys.h: revision 1.81 sys/fs/puffs/puffs_sys.h: revision 1.82 lib/libperfuse/subr.c: revision 1.19 lib/libperfuse/perfuse.c: revision 1.30 sys/fs/puffs/puffs_msgif.c: revision 1.90 sys/fs/puffs/puffs_msgif.c: revision 1.91 sys/fs/puffs/puffs_msgif.c: revision 1.92 lib/libperfuse/ops.c: revision 1.59 lib/libpuffs/puffs.3: revision 1.53 lib/libperfuse/debug.c: revision 1.12 lib/libpuffs/puffs.3: revision 1.54 sys/fs/puffs/puffs_vnops.c: revision 1.167 sys/fs/puffs/puffs_msgif.h: revision 1.79 usr.sbin/perfused/msg.c: revision 1.21 sys/fs/puffs/puffs_vfsops.c: revision 1.102 sys/fs/puffs/puffs_vfsops.c: revision 1.103 sys/fs/puffs/puffs_vfsops.c: revision 1.105 lib/libpuffs/puffs.h: revision 1.123 lib/libperfuse/perfuse_if.h: revision 1.20 lib/libperfuse/perfuse.c: revision 1.29 lib/libpuffs/dispatcher.c: revision 1.42 lib/libpuffs/dispatcher.c: revision 1.43 - Fix same vnodes associated with multiple cookies The scheme used to retreive known nodes on lookup was flawed, as it only used parent and name. This produced a different cookie for the same file if it was renamed, when looking up ../ or when dealing with multiple files associated with the same name through link(2). We therefore abandon the use of node name and introduce hashed lists of inodes. This causes a huge rewrite of reclaim code, which do not attempt to keep parents allocated until all their children are reclaimed - Fix race conditions in reclaim There are a few situations where we issue multiple FUSE operations for a PUFFS operation. On reclaim, we therefore have to wait for all FUSE operation to complete, not just the current exchanges. We do this by introducing node reference count with node_ref() and node_rele(). - Detect data loss caused by FAF VOP_PUTPAGES causes FAF writes where the kernel does not check the operation result. At least issue a warning on error. - Enjoy FAF shortcut on setattr No need to wait for the result if the kernel does not want it. There is however an exception for setattr that touch the size, we need to wait for completion because we have other operations queued for after the resize. - Fix fchmod() on write-open file fchmod() on a node open with write privilege will send setattr with both mode and size set. This confuses some FUSE filesystem. Therefore we send two FUSE operations, one for mode, and one for size. - Remove node TTL handling for netbsd-5 for simplicity sake. The code still builds on netbsd-5 but does not have the node TTL feature anymore. It works fine with kernel support on netbsd-6. - Improve PUFFS_KFLAG_CACHE_FS_TTL by reclaiming older inactive nodes. The normal kernel behavior is to retain inactive nodes in the freelist until it runs out of vnodes. This has some merit for local filesystems, where the cost of an allocation is about the same as the cost of a lookup. But that situation is not true for distributed filesystems. On the other hand, keeping inactive nodes for a long time hold memory in the file server process, and when the kernel runs out of vnodes, it produce reclaim avalanches that increase lattency for other operations. We do not reclaim inactive vnodes immediatly either, as they may be looked up again shortly. Instead we introduce a grace time and we reclaim nodes that have been inactive beyond the grace time. - Fix lookup/reclaim race condition. The above improvement undercovered a race condition between lookup and reclaim. If we reclaimed a vnode associated with a userland cookie while a lookup returning that same cookiewas inprogress, then the kernel ends up with a vnode associated with a cookie that has been reclaimed in userland. Next operation on the cookie will crash (or at least confuse) the filesystem. We fix this by introducing a lookup count in kernel and userland. On reclaim, the kernel sends the count, which enable userland to detect situation where it initiated a lookup that is not completed in kernel. In such a situation, the reclaim must be ignored, as the node is about to be looked up again. Fix hang unmount bug introduced by last commit. We introduced a slow queue for delayed reclaims, while the existing queue for unmount, flush and exist has been renamed fast queue. Both queues had timestamp for when an operation should be done, but it was useless for the fast queue, which is always used to run an operation ASAP. And the timestamp test had an error that turned ASAP into "at next tick", but nobody what there to wake the thread at next tick, hence the hang. The fix is to remove the useless and buggy timestamp test for fast queue. Rename slow sopreq queue into node sopreq queue, to refet the fact that is only intended for postponed node reclaims. When purging the node sopreq queue, do not call puffs_msg_sendresp(), as it makes no sense. Fix race condition between (create|mknod|mkdir|symlino) and reclaim, just like we did it between lookup and reclaim. Missing bit in previous commit (prevent race between create|mknod|mkdir|symlink and reclaim) Bump date for previous. New sentence, new line; remove trailing whitespace; fix typos; punctuation nits. Add PUFFS_KFLAG_CACHE_DOTDOT so that vnodes hold a reference on their parent, keeping them active, and allowing to lookup .. without sending a request to the filesystem. Enable the featuure for perfused, as this is how FUSE works. Missing bit in previous commit (PUFFS_KFLAG_CACHE_DOTDOT option to avoid looking up ..)
- Improve PUFFS_KFLAG_CACHE_FS_TTL by reclaiming older inactive nodes. The normal kernel behavior is to retain inactive nodes in the freelist until it runs out of vnodes. This has some merit for local filesystems, where the cost of an allocation is about the same as the cost of a lookup. But that situation is not true for distributed filesystems. On the other hand, keeping inactive nodes for a long time hold memory in the file server process, and when the kernel runs out of vnodes, it produce reclaim avalanches that increase lattency for other operations. We do not reclaim inactive vnodes immediatly either, as they may be looked up again shortly. Instead we introduce a grace time and we reclaim nodes that have been inactive beyond the grace time. - Fix lookup/reclaim race condition. The above improvement undercovered a race condition between lookup and reclaim. If we reclaimed a vnode associated with a userland cookie while a lookup returning that same cookiewas inprogress, then the kernel ends up with a vnode associated with a cookie that has been reclaimed in userland. Next operation on the cookie will crash (or at least confuse) the filesystem. We fix this by introducing a lookup count in kernel and userland. On reclaim, the kernel sends the count, which enable userland to detect situation where it initiated a lookup that is not completed in kernel. In such a situation, the reclaim must be ignored, as the node is about to be looked up again.
Pull up following revision(s) (requested by manu in ticket #392): lib/libpuffs/puffs_ops.3: revision 1.32 lib/libperfuse/ops.c: revision 1.58 lib/libpuffs/puffs.h: revision 1.122 lib/libpuffs/dispatcher.c: revision 1.41 lib/libperfuse/perfuse_priv.h: revision 1.30 Pass the FAF flag to handlers: - setattr_ttl is updated to add a flag argument. Since it was not present in a previous release, we can change its API - write2 is introduced, this is write with an extra flag for FAF. - fsync already has the FAF information in a flag and needs no change - for other operations, FAF is unconditional Fix the build by adding (unused) flags argument to perfuse_node_setattr_ttl
Pass the FAF flag to handlers: - setattr_ttl is updated to add a flag argument. Since it was not present in a previous release, we can change its API - write2 is introduced, this is write with an extra flag for FAF. - fsync already has the FAF information in a flag and needs no change - for other operations, FAF is unconditional
sync with head.
Pull up following revision(s) (requested by manu in ticket #195): lib/libskey/skeysubr.c: revision 1.27 lib/libkvm/kvm_getloadavg.c: revision 1.11 lib/libwrap/update.c: revision 1.9 lib/liby/yyerror.c: revision 1.9 lib/libpuffs/puffs_ops.3: revision 1.30 lib/libwrap/misc.c: revision 1.10 lib/libwrap/hosts_access.c: revision 1.20 lib/libpuffs/pnode.c: revision 1.11 lib/libperfuse/subr.c: revision 1.17 lib/libpuffs/pnode.c: revision 1.12 lib/libperfuse/subr.c: revision 1.18 lib/libwrap/options.c: revision 1.15 lib/libwrap/fix_options.c: revision 1.11 lib/libperfuse/ops.c: revision 1.52 lib/libperfuse/ops.c: revision 1.53 lib/libperfuse/ops.c: revision 1.54 lib/libwrap/hosts_ctl.c: revision 1.5 lib/libintl/gettext.c: revision 1.27 lib/libwrap/shell_cmd.c: revision 1.6 lib/libpuffs/dispatcher.c: revision 1.39 lib/libperfuse/perfuse_priv.h: revision 1.27 lib/libwrap/socket.c: revision 1.19 lib/libpuffs/puffs.3: revision 1.50 lib/libperfuse/perfuse_priv.h: revision 1.28 lib/libpuffs/puffs_priv.h: revision 1.45 lib/libpuffs/puffs.3: revision 1.51 lib/libperfuse/perfuse_priv.h: revision 1.29 lib/libwrap/percent_x.c: revision 1.5 lib/libpuffs/puffs.3: revision 1.52 lib/libperfuse/debug.c: revision 1.11 sys/fs/puffs/puffs_vnops.c: revision 1.165 lib/libwrap/tcpd.h: revision 1.13 sys/fs/puffs/puffs_vnops.c: revision 1.166 lib/libwrap/eval.c: revision 1.7 sys/fs/puffs/puffs_msgif.h: revision 1.78 sys/fs/puffs/puffs_vfsops.c: revision 1.101 lib/libwrap/rfc931.c: revision 1.9 lib/libwrap/clean_exit.c: revision 1.5 lib/libpuffs/puffs.h: revision 1.120 lib/libc/stdlib/jemalloc.c: revision 1.27 lib/librmt/rmtlib.c: revision 1.26 lib/libpuffs/puffs.h: revision 1.121 sys/fs/puffs/puffs_sys.h: revision 1.79 lib/librumpclient/rumpclient.c: revision 1.48 lib/libwrap/refuse.c: revision 1.5 lib/libperfuse/perfuse.c: revision 1.26 lib/libperfuse/perfuse.c: revision 1.27 tests/fs/puffs/t_fuzz.c: revision 1.5 lib/libperfuse/perfuse.c: revision 1.28 lib/libpuffs/dispatcher.c: revision 1.40 sys/fs/puffs/puffs_node.c: revision 1.24 lib/libwrap/diag.c: revision 1.9 lib/libintl/textdomain.c: revision 1.13 Use C89 function definition Add name and atttribute cache with filesytem provided TTL. lookup, create, mknod, mkdir, symlink, getattr and setattr messages have been extended so that attributes and their TTL can be provided by the filesytem. lookup, create, mknod, mkdir, and symlink messages are also extended so that the filesystem can provide name TTL. Add PUFFS_KFLAG_CACHE_FS_TTL flag to puffs_init(3) to use name and attribute cache with filesystem provided TTL. lookup, create, mknod, mkdir, symlink, getattr and setattr messages have been extended so that attributes and their TTL can be provided by the filesytem. lookup, create, mknod, mkdir, and symlink messages are also extended so that the filesystem can provide name TTL. The filesystem updates attributes and TTL using puffs_pn_getvap(3), puffs_pn_getvattl(3), and puffs_pn_getcnttl(3) Use new PUFFS_KFLAG_CACHE_FS_TTL option to puffs_init(3) so that FUSE TTL on name and attributes are used. This save many PUFFS operations and improves performances. PUFFS_KFLAG_CACHE_FS_TTL is #ifdef'ed in many places for now so that libperfuse can still be used on netbsd-5. Split file system. Comma fixes. Remove dangling "and". Bump date for previous. - Makesure update_va does not change vnode size when it should not. For instance when doing a fault-issued VOP_GETPAGES within VOP_WRITE, changing size leads to panic: genfs_getpages: past eof. -Handle ticks wrap around for vnode name andattribute timeout - When using PUFFS_KFLAG_CACHE_FS_TTL, do not use puffs_node to carry attribute and TTL fora newly created node. Instead extend puffs_newinfo and add puffs_newinfo_setva() and puffs_newinfo_setttl() - Remove node_mk_common_final in libperfuse. It used to set uid/gid for a newly created vnode but has been made redundant along time ago since uid and gid are properly set in FUSE header. - In libperfuse, check for corner case where opc = 0 on INACTIVE and RECLAIM (how is it possible? Check for it to avoid a crash anyway) - In libperfuse, make sure we unlimit RLIMIT_AS and RLIMIT_DATA so that we do notrun out of memory because the kernel is lazy at reclaiming vnodes. - In libperfuse, cleanup style of perfuse_destroy_pn() Do not set PUFFS_KFLAG_CACHE_FS_TTL for PUFFS tests
- When using PUFFS_KFLAG_CACHE_FS_TTL, do not use puffs_node to carry attribute and TTL fora newly created node. Instead extend puffs_newinfo and add puffs_newinfo_setva() and puffs_newinfo_setttl() - Remove node_mk_common_final in libperfuse. It used to set uid/gid for a newly created vnode but has been made redundant along time ago since uid and gid are properly set in FUSE header. - In libperfuse, check for corner case where opc = 0 on INACTIVE and RECLAIM (how is it possible? Check for it to avoid a crash anyway) - In libperfuse, make sure we unlimit RLIMIT_AS and RLIMIT_DATA so that we do notrun out of memory because the kernel is lazy at reclaiming vnodes. - In libperfuse, cleanup style of perfuse_destroy_pn()
sync with head
Add PUFFS_KFLAG_CACHE_FS_TTL flag to puffs_init(3) to use name and attribute cache with filesystem provided TTL. lookup, create, mknod, mkdir, symlink, getattr and setattr messages have been extended so that attributes and their TTL can be provided by the filesytem. lookup, create, mknod, mkdir, and symlink messages are also extended so that the filesystem can provide name TTL. The filesystem updates attributes and TTL using puffs_pn_getvap(3), puffs_pn_getvattl(3), and puffs_pn_getcnttl(3)
Use __dead
Pull up following revision(s) (requested by manu in ticket #1645): lib/libc/sys/Makefile.inc 1.207 via patch lib/libc/sys/extattr_get_file.2 patch lib/libpuffs/dispatcher.c 1.34,1.36 via patch lib/libpuffs/puffs.c 1.107 via patch lib/libpuffs/puffs.h 1.115,1.118 via patch sys/fs/puffs/puffs_msgif.h 1.71,1.76 via patch sys/fs/puffs/puffs_vfsops.c 1.88 via patch sys/fs/puffs/puffs_vnops.c 1.145,1.154 via patch sys/kern/vfs_xattr.c 1.24-1.27 via patch sys/kern/vnode_if.c 1.87 via patch sys/sys/Makefile 1.133 via patch sys/sys/extattr.h 1.6 via patch sys/sys/vnode_if.h 1.81 via patch sys/ufs/ffs/ffs_vnops.c patch sys/ufs/ufs/ufs_extattr.c 1.31,1.34 via patch * support extended attributes * bump major due to structure growth * add some spare space * remove ABI sillyness Support extended attributes. Fix multiple non compliances in our Linux-like extattr API, and make it public so that it can be used. Improve a bit listxattr(2). It attemps to list both system and user extended attributes, and it faled if calling user did not have privilege for reading system EA. Now we just lise user EA and skip system EA in reading them is not allowed. Fix bug introduced in previous commuit: Do not vrele() a vnode we did not obtained. Improve UFS1 extended attributes usability - autocreate attribute backing file for new attributes - autoload attributes when issuing extattrctl start - when autoloading attributes, do not display garbage warning when looking up entries that got ENOENT Add a flag to VOP_LISTEXTATTR(9) so that the vnode interface can tell the filesystem in which format extended attribute shall be listed. There are currently two formats: - NUL-terminated strings, used for listxattr(2), this is the default. - one byte length-pprefixed, non NUL-terminated strings, used for extattr_list_file(2), which is obtanined by setting the EXTATTR_LIST_PREFIXLEN flag to VOP_LISTEXTATTR(9) This approach avoid the need for converting the list back and forth, except in libperfuse, since FUSE uses NUL-terminated strings, and the kernel may have requested EXTATTR_LIST_PREFIXLEN.
Add a flag to VOP_LISTEXTATTR(9) so that the vnode interface can tell the filesystem in which format extended attribute shall be listed. There are currently two formats: - NUL-terminated strings, used for listxattr(2), this is the default. - one byte length-pprefixed, non NUL-terminated strings, used for extattr_list_file(2), which is obtanined by setting the EXTATTR_LIST_PREFIXLEN flag to VOP_LISTEXTATTR(9) This approach avoid the need for converting the list back and forth, except in libperfuse, since FUSE uses NUL-terminated strings, and the kernel may have requested EXTATTR_LIST_PREFIXLEN.
Make retval argument for pathconf a register_t to match VOP_PATHCONF. This makes the size the same on 64bit archs. Don't bother bumping any version, since you'd have explicitly had to jump through some hoops to use pathconf before.
add option string for no attribute cache (foreseeing the odd event I might actually implement one some day)
* support extended attributes * bump major due to structure growth * add some spare space * remove ABI sillyness
sync to netbsd-5
Add convenience routine puffs_unmountonsignal(), which does exactly that.
Add an error handler which syslogs but does not abort() and make it the default. Most servers just shovel bits from their backend to the kernel without checking the contents and dumping core in case the backend gives garbage is a bit harsh.
Pull up following revision(s) (requested by pooka in ticket #1105): lib/libpuffs/puffs.h: revision 1.112 lib/libpuffs/puffs.c: revision 1.100 distrib/sets/lists/base/md.amd64: patch distrib/sets/lists/base/md.sparc64: patch distrib/sets/lists/base/shl.mi: patch lib/libpuffs/shlib_version: patch Make it possible to supply parameters to puffs_init() later (but before puffs_mount()).
Make it possible to supply parameters to puffs_init() later (but before puffs_mount()).
Pull up the following revisions(s) (requested by pooka in ticket #1100): lib/libpuffs/dispatcher.c: revision 1.33 lib/libpuffs/puffs.c: revision 1.99 lib/libpuffs/puffs.h: revision 1.111 sys/fs/puffs/puffs_msgif.h: revision 1.67 via patch sys/fs/puffs/puffs_vnops.c: revision 1.136 Support VOP_ABORTOP() in puffs.
Dispatch puffs_node_abortop(). Note: We use the storage of puffs_cache_write from puffs_ops for this purpose. It's not issued by the kernel and hence currently unused, and this saves us from the trouble of bumping the lib major version.
Pull up the following revisions(s) (requested by pooka in ticket #1095): lib/libpuffs/puffs.c: revision 1.94 lib/libpuffs/puffs.h: revision 1.110 Add puffs_cancel(), which allows to back down after puffs_init() and before puffs_mount().
Pull up the following revisions(s) (requested by pooka in ticket #1094): lib/libpuffs/puffs.3: revision 1.45 lib/libpuffs/puffs.c: revision 1.93 lib/libpuffs/puffs.h: revision 1.109 Add function puffs_setspecific(pu, private) which can be used to set the specific data after the call to puffs_init().
Add puffs_cancel(), which allows to back down after puffs_init() and before puffs_mount(). (this should actually be integrated with puffs_exit(), but that routine is currently so non-sensible that I didn't bother now)
add:
puffs_setspecific(pu, private)
Can be used to set the specific data after the call to
puffs_init().
Sync with wrstuden-revivesa-base-2.
Replace void * by puffs_cookie_t where appropriate. No functional change.
sync with HEAD
Fix a weirdosity in the framebuf gotfb interface: hand buffer ownership to the called routine instead of keeping it. This means that callees must now destroy the buffer once they are done with it.
Bump version for lazy contexts (probably not necessary, but it's been a long time since I bumbed the version ;)
Implement lazy contexts for file systems using puffs_mainloop(). Previously each request was executed on its own callcontext and switched to every time the request was being processed. Now requests borrow the mainloop context and switch only if/when they yield. This takes two context switches away from a file system request bringing down the typical amounts 2->0 (e.g. dtfs) and 4->2 (e.g. psshfs). The interfaces for manually executing requests changed a bit: puffs_dispatch_create() and puffs_dispatch_exec() must now be used. They are not tested, as nothing in-tree wants them and I doubt anyone else is really interested in them either. Also do some misc code cleanup related to execution contexts. The "work-in-progress checkpoint" committed over a year ago was starting to look slightly weed-infested.
Introduce PUFFS_STACKSIZE_MIN to avoid overloading 0 and getting a warning from the library when really wanting to say "just the minimum, please".
sync with HEAD
Add a public interface for setting a cookie mapper.
Add generic getattr routine.
* nuke puffs_cc_get{specific,usermount} for good
* move prototypes for puffs_docc and puffs_dopufbuf into the
public header, as they are should be exposed
nuke #define whose purpose has ceased to be
Now that "l" is gone both as an argument to operations and from componentname, remove all vestiges of puffs_cid.
Part 3/n of extensive changes to message handling: Get rid of the original puffs_req(3) framework and use puffs_framebuf(3) instead for file system requests. It has the advantage of being suitable for transporting a distributed message passing protocol and therefore us being able to run the file system server on any host. Ok, puffs is not quite here yet: libpuffs needs to grow request routing support and the message contents need to be munged into a host independent format. Saying which format would be telling, but it might begin with an X, end in an L and have the 13th character in the middle. Keep an eye out for the sequels: Parts 3+m/n.
Rototill. Ok, ok, a few more words about it: stop holding puffs_cc as a holy value and passing it around to almost every possible place (popquiz: which kernel variable does this remind you of?). Instead, pass the natural choice, puffs_usermount, and fetch puffs_cc via puffs_cc_getcc() only in routines which actually need it. This not only simplifies code, but (thanks to the introduction of puffs_cc_getcc()) enables constructs which weren't previously sanely possible, say layering as a curious example. There's still a little to do on this front, but this was the major fs interface blast.
* allocate pcc as a slice from the stack instead of allocating it separately * provide puffs_cc_getcc() This is in preparation for the removal of you-should-guess-what as an argument to routines here and there and everywhere.
Remove "puffs_cid" from the puffs interface following l-removal from the kernel vfs interfaces. puffs_cc_getcaller(pcc) can be used now should the same information be desired.
Introduce puffs_daemon(), which works like daemon(3) except that it also synchronizes with puffs_mount() and does not return (exit) in the parent process until the file system has been mounted. This makes it possible to reliably run e.g. mount_foo jippi /kai ; cd /kai/ee
sync with HEAD
remember to bump version for puffs_mainloop() change
Pull the daemonizing code out of the library mainloop into the file servers. Calling daemon() (i.e. fork()ing) inside a library can cause nice surprises for e.g. threaded programs. As discussed with Greg Oster & others.
Add possibility to set pre- and post callbacks which will be called for all operations before or after executing the specific callback (the api is not be final yet, though).
mmap stacks, makes it possible to align them
Add puffs_cc_schedule() which marks a pcc runnable and will pass execution to it when in the main loop the next time.
Always provide caller information from the kernel based on curlwp. (but don't deprecate the old puffs_cid interface just yet)
* Change type of boolean functions to, well, boolean instead of int. * add puffs_cid_isequal() which can be used to test if the two caller id's refer to the same process/lwp.
Part 1/n of some pretty extensive changes to how the kernel module
interacts with the userspace file server:
* since the kernel-user communication is not purely request-response
anymore (hasn't been since 2006), try to rename some "request" to
"message". more similar mangling will take place in the future.
* completely rework how messages are allocated. previously most of
them were borrowed from the stack (originally *all* of them),
but now always allocate dynamically. this makes the structure
of the code much cleaner. also makes it possible to fix a
locking order violation. it enables plenty of future enhancements.
* start generalizing the transport interface to be independent of puffs
* move transport interface to read/write instead of ioctl. the
old one had legacy design problems, and besides, ioctl's suck.
implement a very generic version for now; this will be
worked on later hopefully some day reaching "highly optimized".
* implement libpuffs support behind existing library request
interfaces. this will change eventually (I hate those interfaces)
Pass reason string to pu_errnotify. Print it in the default version before aborting.
Add error notifications, which are used to deliver errors from the kernel to the file server for silly things the file server did, e.g. attempting to create a file with size VSIZENOTSET. The file server can handle these as it chooses, but the default action is for it to throw its hands in the air and sing "goodbye, cruel world, it's over, walk on by".
Give cmpframe() the ability to shortcircuit the comparison loop by signalling back that the buffer under examination is not a response at all.
Add gotframe callback, which is called if an incoming frame matches no outstanding request.
Mimic namei structure changes for puffs. bump both kernel & lib version.
puffs_cc_create/destroy aren't supposed to be exported to the user
Add accessors for struct puffs_node to prepare for removal from exported headers.
Nuke PUFFSLOOKUP_FOO and move to NAMEI_FOO
mmap fflags -> prot
file puffs.h was added on branch matt-mips64 on 2007-07-27 08:28:17 +0000
mmap fflags -> prot
Add puffs_framev_enqueue_waitevent, which waits for read/write/error events from an i/o descriptor. It yields until an event happens and continues execution when the event loop notices the event has happened.
* add mntfromname parameter to puffs_init() * set it in file servers
add IO_NDELAY
g/c leftover
make notifyfunction optional and do not call removeonclose() if none is given.
Make puffs_framebuf better suited from environments with multiple sources: * in addition add/remove, allow enable/disable, which can be used to control events for descriptors without having to remove all the data associated with them * add directsend/receive, which can be used to pass the same buffer from the caller to read/writeframe and back again * add flags to enqueue functions and allow urgent buffers to be processed as the next PDU
Get rid of the "int *refs" parameter to inactive: the same can be accomplished now with puffs_setbacks.
Instead of requesting various pieces of information from node-creating methods using "please fill me"-pointers, pass "struct puffs_newinfo *" and provide methods to fill in new node info.
Instead of supplying a plain pid, supply an abstract struct puffs_cid *, which can currently be used to query the pid and lwpid.
make puffs_cred an opaque type
Actually, keep PUFFS_KFLAG_NOCACHE and -o cache around as shorthand to neither page- nor namecache.
PUFFS_KFLAG_NOCACHE became two, so introduce the command line options -o nonamecache and -o nopagecache and adapt file systems where necessary.
Introduce puffs_null_setops(), which initializes the ops vector with puffs nullfs ops.
Introduce puffs_pn_remove() to signal that a node has been deleted and destroyed, but not yet reclaimed. This prevents puffs_pn_nodewalk() from returning stale entries. Make nullfs use this (some file systems are a bit too happy with recycling inode numbers).
Move puffs to a two clause license where it already isn't so. And as agc pointed out, even files with the third clause were already effectively two clause because of a slight bug in the language...
Introduce puffs_framev_framebuf_ccpromote, which can be used to turn a reference to puffs_framebuf in the file system from a cb/justsend operation to a cc wait, should the file system find itself desiring the result.
Change the signature of puffs_respcmp to return -1,0,1 for smaller, equal, larger, respectively instead of 0/1 for non/equal. This will allow sorting the buffers for faster matching in libpuffs. While here, change the name from respcmp to framecmp, as that better reflects the purpose. NOTE! there is no obvious way to make compilation fail for file systems which may already be using this feature (although I don't think there are any outside our tree, as the feature is two weeks old). Nevertheless, non-updated file systems will fail very quickly.
dispatch PUFFS_VN_POLL
Reflect kernel changes for supplying root node information in mount instead of puffs_start(). Get completely rid of puffs_start(), as everything it used to do is now handled by the mount routine. Introduce an optional pre-mount call puffs_setrootinfo() for setting non-default root node information. As the old puffs_mount() is now virtually useless, say byebye to it and rename the old puffs_domount() to puffs_mount(), but add a root cookie parameter to compensate for the late puffs_start().
Add error parameter to puffs_framev_cb to signal if there is a result of if the function should just do resource cleanup.
Step 3 and currently final step of a more general purpose event loop: support removal and addition of i/o file descriptors on the fly. * detect closed file descriptors * automatically free waiters of a dead file descriptor * give the file server the possibility to specify a callback which notifies of a dead file descriptor * move loop function to be a property of the mainloop instead of framebuf (doesn't change effective behaviour) * add the possibility to configure a timespec parameter which attempts to call the loop function periodically * move the event loop functions from the puffs_framebuf namespace to puffs_framev to differential between pure memory management functions
Step 2 of general-purpose eventloop: get rid of puffs_framebuf_eventloop(). puffs_mainloop() now handles that functionality as well.
Step 1 in moving towards a general-purpose eventloop: allow the framebuf event loop to take n i/o fd's as parameters and also allow dynamic add/remove of fd's. (not tested except for one fd, but more changes coming soon)
Remove the puffs_usermount argument from puffs_req_handle(). It can be found from the other arguments and this way puffs_req_handle() is more symmetric with the other puffs_req() routines.
Add puffs_setback(), which can be used to set setbacks for nodes before the call returns to the kernel. Currently it can be called only from open(), mmap(), remove() and rmdir(). (But that might change)
Remove the user-specified argument for the framebuf loop callback. It can be generated through other means if it is desired.
Add puffs "frame buffers", which are an abstraction of the buffering and event handling mechanisms required in file servers with blocking I/O backends. puffs_framebuf is built on the concept of puffs_cc and uses those to multiplex execution where needed. File systems are required to implement three methods: * read frame * write frame * compare if frame is a response to the given one Memory management is provided by puffs_framebuf, but the file systems must still, of course, interpret the protocol and do e.g. byte order conversion. As always, puffs_framebuf is work in progress. Current users are mount_psshfs and mount_9p.
Introduce PUFFS_FLAG_HASHPATH, which for PUFFS_FLAG_BUILDPATH file servers creates a hash of the node's path into po->po_hash to avoid a full pathcmp every time.
use more accurate flag masking
add puffs_cc_getspecific(pcc), which is shorthand for: puffs_getspecific(puffs_cc_getusermount(pcc))
Introduce PUFFS_STORE_DCOOKIE, which stores a directory cookie for a given offset iff the directory read in question wants cookies.
alloc to set flags when setting fhsize
some routines for setting mount parameters
Instead of keeping on adding parameters to puffs_mount(), make it only take the bare essentials, which currently means removing "maxreqlen" from the argument list (all current callers I'm aware of set it as 0 anyway). Introduce puffs_init(), which provides a context for setting various parameters and puffs_domount(), which can be used to mount the file system. Keep puffs_mount() as a shortcut for the above two for simple file systems. Bump development ABI version to 13. After all, it's Friday the 13th. Watch out! Bad things can happen on Friday the 13th. --No carrier--
Hide struct puffs_usermount from the rest of the world and provide accessors for interesting data in it. Namely, you can now get pu->pu_privdata with puffs_getspecific(), pu->pu_pn_root with puffs_set/getroot() and pu->pu_maxreqlen with puffs_getmaxreqlen().
* support nodetofh and fhtonode callbacks * use new signature of node_readdir
support page cache flushing
implement puffs_inval_pagecache_node_range()
unify the location of *pcr in puffs_access_*
* return -1 and set errno instead of returning errno directly * introduce puffs_cred_isregular() which checks if the cred structure has regular uid/gid (i.e. "person" credentials and not kernel creds)
remove variable names from header
add routiens which check against ufs semantics for permission to chown, chmod and change a/mtime
implement counterpart of vaccess()
initial support for cacheops
va->va_mode doesn't contain the full argument to the mknod() system call, so introduce puffs_addvtype2mode() and use that in null.c to generate the proper syscall argument
Introduce puffs_path_walkcmp(), which is to be called from nodewalk and compares the path of the node against the given pathobject. Also make comparison method take a flag to indicate if it should check if the second path is a true prefix of the first. plus some namespace cleanup
* get rid of the holy kernel-given ISDOTDOT, and instead define a macro which does strcmp against ".." and (the untranslated) componentname * make PUFFS_FLAG_BUILDPATH build paths also if dotdot is the case, and adapt the regular path objects to this * make nullfs lookup readable because we can now get rid of dotdot processing there
add interfaces for suspending the file system
Be consistent with the rest of the framework about the argument order of puffs_docc and puffs_dopreq
be consistent in naming request handlers
revoke revoke
Checkpoint some more work in progress: for the benefit those file system backends which operate purely based on paths, push out more path management into the library and make path management more abstract: enable a file system to define a bunch of path management callbacks, which are used by the framework. Management of normal /this/is/a/path type paths is provided by the library.
implement fsync
Add preliminary code for a nullfs layer, which can be used to mount a directory hierarchy to another point, just like with the kernel nullfs. This is not really a layering scheme yet, but it should evolve into one. Currently it can just be used to do 1:1 mapping.
Be less shy about the default stack size, it's a only virtual memory space. TODO: put a guard page at the end
rename name cache invalidation interface to be in sync with the kernel operation names, i.e. contain "namecache" instead of just "name"
lib interface for invalidation routines
* get rid of the mount callback; it's no great surprise to the server that it needs to mount the file system backend if it wants to call mount * provide some options for getmntopts(), assume that callers will parse command line (or fstab) args * reorganize the puffs_cc interface just a bit, preparing for a bigger revamp later
* inform kernel of our version * homegrown lib ABI check (to avoid bumping major a ludicrillion times before reaching stability)
checkpoint some experimental work-in-progress, namely: Add support for having multiple outstanding operations. This is done by exposing enough interfaces so that it is convenient to have the main event loop in the implementation itself and by providing a continuation framework for convinient blocking and rescheduling. works fine, but will undergo further cleanup & development
outline & export code for interfacing with kernel GET/PUT requests
support mmap
rototill the operation callbacks: unify the ops under struct puffs_ops and namespace them
PUFFSFLAG -> PUFFS_FLAG to be consistent with the kernel
summon daemon(3) in mainloop unless the nodaemon flag is given
const coherence
mode_t to enum vtype conversion
* call statvfs() at mount() time, pass that info along * start is gone
shuffle flags a bit
namespace previous change under puffs_vfsnop to avoid confusions
implement unmount, sync and statvfs as dummies for file system which couldn't care less about implementing them
* catch up with kernel changes * better error handling when mounting
* support FAF (Fire-And-Forget) class operations * adapt to new lookup
pass rdev for specfs support
handle PUFFS_VN_INACTIVE
add libpuffs - the userspace programming interface for puffs This work was initially started and completed for Google SoC 2005 and tweaked to work a bit better in the past few weeks. While being far from complete, it is functional enough to be able and stable to host a fairly general-purpose in-memory file system in userspace. Even so, puffs should be considered experimental and no binary compatibility for interfaces or crash-freedom or zero security implications should be relied upon just yet. The GSoC project was mentored by William Studenmund and the final review for the code was done by Christos.