Up to [cvs.NetBSD.org] / pkgsrc / mk / install
Request diff between arbitrary revisions
Keyword substitution: kv
Default branch: MAIN
mk/install: New install-env target. Works like configure-env, build-env, and test-env.
mk: Add mktool support to install-ctf. mktool is an optional external tool, available from pkgtools/mktool or "cargo install mktool", that provides significant performance improvements compared to the default install-ctf target, while being 100% compatible. After installing, set TOOLS_PLATFORM.mktool=/path/to/mktool to enable it. mktool allows ctfconvert processes to run in parallel, significantly reducing build time, especially in packages that ship lots of large binaries. During testing, enabling mktool reduced the install-ctf time in emulators/qemu on a SmartOS host from 40m 39s to 5m 13s.
mk: Skip rust .rlib archives for debug stripping.
mk: Add a few more common _DEBUG_SKIP_PATTERNS.
mk: install performance improvements.
Inline ${RUN} calls where appropriate. Only call mkdir and rm when
necessary. Remove useless use of cat(1).
Saves at least 12 execs per build, but often a lot more depending on the
number of files that were not stripped or had CTF conversion applied.
mk: Skip many more CTF and debug files by default. Leave CTF_FILES_SKIP and STRIP_FILES_SKIP purely for packages to set any additional paths to skip.
mk/install: fix indentation and alignment, no functional change
mk/install: fix documentation and speed up install-dirs-from-PLIST The documentation did not mention that conditional entries in PLIST files are skipped. Pkglint doesn't know this either and issues wrong notes that some directories are redundant in INSTALLATION_DIRS when in reality they need to be listed there. While here, filter out some duplicate directories. This invokes the command for creating a single directory less often. Since PLIST files are usually sorted, files in the same directory tend to be listed near each other. The fallback to the plain 'uniq' is needed for SCO_SV, which does not define TOOLS_PLATFORM.uniq.
mk: Skip unnecessary dirs for CTF/debug conversion. There won't (or at least should never!) be any files under share/ or man/ that require conversion for CTF or debug support, so set sensible defaults for both CTF_FILES_SKIP and STRIP_FILES_SKIP. Further additions are welcome. While here rearrange the ordering of the debug skips to match CTF and deliver a small performance improvement by avoiding unnecessary file tests. Combined, these reduce the runtime for "make install-ctf install-strip-debug" in lang/rust down from wall/user/sys 10m33s/2m34s/9m30s to 1m13s/0m46s/1m4s.
mk: Skip CTF_FILES_SKIP before anything else.
mk: use a single form for headings in the documentation comments
mk: allow "bmake clean depends" as shortcut When "bmake clean depends" was called for a package where the various cookie files already existed, these would enable different rules than a clean package directory. Since "bmake clean" deletes all the cookie files before "bmake depends" starts, in these combined command lines the cookie files must be treated as absent.
mk: Add CTF infrastructure support. Platform support is determined by _OPSYS_SUPPORTS_CTF from mk/platform, the user enables support by setting PKGSRC_USE_CTF=yes, and packages can explicitly disable support with CTF_SUPPORTED=no or skip certain files with CTF_FILES_SKIP. The path to ctfconvert is configured via TOOLS_PLATFORM.ctfconvert. If all of the requisite variables are enabled, a compiler-specific debug flag is passed via the wrappers to ensure we have DWARF information to convert, _INSTALL_UNSTRIPPED is explicitly defined to avoid binaries being stripped prior to conversion, and the conversion is performed during the install stage. It is recommended that users who enable the feature also set STRIP_DEBUG=yes to reduce the final binary size once the conversion has been performed. This has been used for the past year in Joyent SmartOS builds. FreeBSD is marked as supported but is untested.
mk/install: Add support for STRIP_DEBUG_SUPPORTED. This is a package-settable variable that will disable stripping binaries if set to anything other than "yes" (the default). This helps packages such as anything built using golang where stripping binaries is harmful to them. Document this and the recently introduced STRIP_FILES_SKIP.
mk/install: Improve the strip-debug target. Use the new strip-dbg tool, ensuring that we don't fail when the native strip doesn't support -g. Actually check for the existence of the output file before trying to move it into place. Add support for STRIP_FILES_SKIP for certain files which should not be stripped. And finally, improve performance slightly by skipping symlinks.
Check, don't assert, that INSTALLATION_DIRS are all in ${PREFIX}.
Change the assertion into an install-time check that the paths
listed in INSTALLATION_DIRS are relative paths or are in ${PREFIX}.
This delays the check so that some common variables used when
listing directories in INSTALLATION_DIRS are fully-expanded.
Assert INSTALLATION_DIRS begin with ${PREFIX} or are relative.
Add an assertion that all paths listed in INSTALLATION_DIRS must
either begin with ${PREFIX}/ or are relative paths. Clarify in the
documentation that INSTALLATION_DIRS can list either type of path.
Some packages are using INSTALLATION_DIRS to create directory trees
outside of ${PREFIX}, e.g., under ${PKG_SYSCONFBASE} or ${VARBASE}.
However, these directories are only created as part of the
"install" target, which causes them to only be created during the
staged installation into ${DESTDIR} and not at all if their binary
packages are installed. These packages should be fixed to use
MAKE_DIRS or OWN_DIRS to create these directory trees at the proper
time.
Provide switch to activate new frameworks to replace "pkginstall".
If ${_USE_NEW_PKGINSTALL} is "yes", then use the new "pkgtasks",
"init", and "pkgformat/pkg"-scripts frameworks instead of the
existing "pkginstall" framework.
Support NetBSD man/html entries. Fixes deforaos-*.
Remove support for USE_DESTDIR=no.
Support compressed manual pages with 3am and 3f suffixes.
Modify mk/plist/plist-gnu.mk to convert gnu/man to ${PKGGNUDIR}${PKGMANDIR}.
Also modify mk/install/install.mk to make the conversion when using
AUTO_MKDIRS.
Introduce PKGGNUDIR as a way to control gnu prefix. By default pkgsrc uses LOCABASE/gnu as a prefix for packages to install native versions of GNU tools, which are them symbolically linked back to the 'g' versions of the files in LOCALBASE, and users can then add LOCALBASE/gnu/bin to PATH to pick up those tools. On systems where the GNU environment is desired, PKGGNUDIR now allows users to install the non-'g' files directly into LOCALBASE, making them the default without having to alter PATH, whilst retaining the 'g' files in order to ensure dependencies and tool paths remain the same.
fix typo in comment
Rename "flavor" to "pkgformat". This is from Anton Panev's GSoC 2011 project to add RPM and DPKG support to pkgsrc. (I am not adding that further support in this commit.) This is just a rename of the existing functionality. Now it will be easy to test the GSoC work by simply putting in a single directory (such as "rpm" or "deb"). See http://addpackageforma.sourceforge.net/ for some details. This is from Anton's CVS, but I made some minor changes: - changed plural pkgformats to singular pkgformat (to be consistent) - fixed a few places (in comments) that were missed - catch up on some additions to flavor not in the pkgforma cvs: PKGSRC_SETENV and _flavor-destdir-undo-replace and undo-destdir-replace-install.
Introduce PKGSRC_SETENV, defaulting to SETENV. Can be set to ${SETENV} -i
to santise environment
remove quote from "DESTDIR".
Introduce DESTDIR_VARNAME package-settable variable.
A variable name that should be set as staged installation location
presented as ${DESTDIR} at installation phase.
"DESTDIR" is set by default.
for packages using different variable for staged installation prefix,
and/or using DESTDIR variable for different purpose.
Rename internal package-clean target to package-eat-cookie.
Rename the install phase to stage-install. Introduce a new install target that defaults to either stage-install or package-install, depending on whether DESTDIR support is active and supported by the package or not.
Use the correct variable names.
Fix syntax. Mea culpra
Deal with strip idiosyncracies: - sometimes creates the temporary file, but fails - sometimes it is successful, but still keeps the temporary file - sometimes it crashes when called with more than one argument Bite the bullet and call it individually with explicit output file and clean up after it.
Use filtered plist for install-strip-debug and install-man-handling, simplifying the latter.
Add STRIP_DEBUG option, which removes debug information from all files, leaving basic symbol tables intact. Document that INSTALL_UNSTRIPPED doesn't work for all packages.
Documented INSTALL_UNSTRIPPED.
Honor PKGLOCALEDIR for AUTO_MKDIRS.
Added compatibility code for INSTALLATION_DIRS_FROM_PLIST. I had intended that all the time, but didn't test enough. :( Thanks to wiz for catching this one.
Added AUTO_MKDIRS as an alias for INSTALLATION_DIRS_FROM_PLIST. The
variable is much more comfortable than specifying all the directories in
INSTALLATION_DIRS that using such a long name for it feels like a
punishment. It also doesn't look nice in the package Makefiles, for
example:
old:
INSTALLATION_DIRS= bin ${PKGMANDIR}/man1 ${PKGMANDIR}/man8
new:
AUTO_MKDIRS= yes
The variable INSTALLATION_DIRS_FROM_PLIST will be deprecated slowly.
Removed _INTERACTIVE_COOKIE, after asking on tech-pkg if anyone needs it. It had only been used to change the "nerrors" variable in bulk builds, but this variable is not used further anyway.
It has always annoyed be that package authors had to write ${PKGMANDIR}/
in the definition of INSTALLATION_DIRS, where a short man/ would be much
shorter. Since we already have that man-transforming magic in some other
places of pkgsrc, it's also here.
While here, documented INSTALLATION_DIRS and
INSTALLATION_DIRS_FROM_PLIST so they are found by "bmake help".
Remove bootstrap-register, bootstrap was changed a while ago to use normal building and installation.
Fix a number of bugs in the DESTDIR support:
- check-perms.mk was not DESTDIR aware, prefix files before passing it
to the directory extraction
- PKG_FILELIST_CMD was calling pkg_info, which is fine for normal
installation, but fails of course for DESTDIR. Just drop the @ lines
from ${_DEPENDS_PLIST} and use that.
- To make he former work, ensure that _flavor-generate-metadata is part
of _INSTALL_ALL_TARGETS. It was normally a dependency of
_flavor-register, but that is skipped for DESTDIR.
- Remove ${_DEPENDS_PLIST} when running install-clean.
Remove ${WRKDIR}/.pkgdb during "install-clean" as that directory
contains files that are generated during the "install" phase. This
should fix the problem where PLIST modifications were ignored if the
PLIST was modified between a "make deinstall" and a "make reinstall".
Remove USE_MTREE support. Keep one copy (the NetBSD version) for the purpose of print-PLIST in plist/common-dirs.mtree. Discussed with wiz@, no objections on tech-pkg@.
Check if INSTALLATION_DIRS_FROM_PLIST is defined before using .if !empty... Reported by Mihai Chelaru.
Drop NO_MTREE by making it the default. Introduce USE_MTREE to get the old behaviour back. Discussed on tech-pkg@.
Fixed pkglint warnings.
Replaced the remaining PKG_SILENT/PKG_DEBUG pairs with RUN.
Fixed the quoting in INSTALLATION_DIRS and install-dirs-from-PLIST. In install-dirs-from-PLIST, all directories that don't contain an '@' or '$' character are automatically installed.
Added MAKEFLAGS to the RECURSIVE_MAKE call. Almost all other calls already have it.
Renamed some more flavor-specific targets to have the form _flavor-*. Split check-vulnerable into a general an a flavor-specific part, for consistence with all the other targets.
Make MTREE_FILE and MTREE_ARGS internal by prefixing them with '_'. They are not supposed to be set by the user.
Desupport unused FORCE_PKG_REGISTER.
Remove support for obsolete and unused NO_PKG_REGISTER.
Moved the code that installs check-files on the hook from install.mk to check-files.mk, for two reasons: 1. It belongs there. 2. Now check-files is run before check-perms and check-interpreter.
Removed the tests for CHECK_INTERPRETER and CHECK_PERMS. They are now in ../checks/*.mk.
Moved the check for whether check-wrkref should be run from install/install.mk to check/check-wrkref.mk. Renamed the check-wrkref target to _check-wrkref.
Added some documentation. Since hooks should not be overwritten, removed the !target() check. That way, one gets a warning when attempting to add some code to the hook.
Moved the check whether check-shlibs should be run from install/install.mk to check/check-shlibs.mk. Renamed check-shlibs to _check-shlibs. Since it had been declared as "private" before, this will not break anything.
Just make the PREFIX with INSTALL_DATA_DIR. Drop the conditional and don't use mkdir, as the latter uses the group permissions of the parent directory under BSDs, resulting in group leaks for USE_DESTDIR=yes. Discussed with agc@.
When a package sets INSTALLATION_DIRS_FROM_PLIST to "yes", it does not need to specify INSTALLATION_DIRS itself. Instead, the list is generated from the static PLIST files. Like for INSTALLATION_DIRS, mtree is disabled when this variable is set.
Fix thinko: INSTALLATION_DIRS entries don't have ${DESTDIR} as prefix,
just ${PREFIX}.
Change INSTALL_MAKE_FLAGS to be _appended_ to MAKE_FLAGS when running
"make install". That way, packages don't have to say:
INSTALL_MAKE_FLAGS+= ${MAKE_FLAGS} foo=bar,
instead it suffices to say
INSTALL_MAKE_FLAGS+= foo=bar,
which is also more appropriate since multiple files (Makefile,
Makefile.common, options.mk, *.mk) don't need to care about whether
INSTALL_MAKE_FLAGS already includes MAKE_FLAGS or not.
Note: It is no longer possible to override MAKE_FLAGS completely. But
except for some exotic platforms, the default value is empty anyway.
Added a new post-installation check to see whether the files have their permissions properly set. See mk/check/check-perms.mk and sysutils/checkperms for details.
Main infrastructure for DESTDIR support.
Packages may set PKG_DESTDIR_SUPPORT to either "destdir" or
"user-destdir" to flag support for this, following the same
rules as PKG_INSTALLATION_TYPES (e.g. define before first include
of bsd.prefs.mk).
The user activates it via USE_DESTDIR. When set to "yes",
packages with "user-destdir" are handled as "destdir".
The installation of the package will not go to ${LOCALBASE},
but a subdirectory of ${WRKDIR} instead. pre/post install scripts are
not run and the package is not registered either. A binary package
can be created instead to be installed normally with pkg_add.
For "user-destdir" packages, everything is run as normal user and
ownership is supposed to be correctled by pkg_create later. Since
the current pkg_install code uses pax and it doesn't allow overwriting
owners, this does not work yet.
For "destdir" packages, installation, packaging and cleaning is run as
root.
This commit does not change the handling of DEPENDS_TARGET or
bin-install to allow recursive usage.
Add variable to control whether the install target is run as root.
Added more error checking: If a directory is specified in INSTALLATION_DIRS but a regular file of the same name already exists, the BSD install(1) program doesn't care but exits successfully. To avoid much more confusing error messages, this is checked here, and a _good_ error message is printed.
Both, the pkgsrc user and the package maintainer must agree that this package should run the CHECK_FILES test. Otherwise we cannot prevent the broken shells/standalone-tcsh from scanning the whole filesystem. (hi agc!)
Rename variable MAKEFILE to MAKE_FILE, as suggested in PR 28392.
Only acquire the localbase lock after acquiring root privileges. This fixes "make install" using just-in-time su-to-root.
Fixed the locking, as suggested by Johnny on the tech-pkg mailing list. Added two targets acquire-localbase-lock and release-localbase-lock, which mark the complete LOCALBASE directory as locked, so that multiple packages cannot run the install, deinstall or bin-install targets at the same time. The install target aquires locks in both WRKSRC and LOCALBASE, the other two targets only need the LOCALBASE lock, since they may be run without WRKSRC being present on the system. locking.mk must be included before tools.mk and the PKG_FAIL_REASON check.
Whenever we invoke a recursive make, we need to ensure that the proper
environment ${PKGSRC_MAKE_ENV} is also passed along. Create a
convenience variable RECURSIVE_MAKE that does exactly this and that
can be used in place of MAKE when invoking make recursively.
Use RECURSIVE_MAKE everywhere in pkgsrc/mk that we invoke make
recursively.
Modify the barrier so that we always invoke the recursive make process when passing through the barrier. This ensures the PATH (passed via PKGSRC_MAKE_ENV) is correctly set for all phases after the barrier. This fixes a bug in "interactive" pkgsrc use, where if you have no work directory and type "make build && make install", then the "install" step does not have a PATH set to include all the wrapper and tools directories.
The package directory path is "PKGPATH", not "PKG_PATH", which is used by pkg_add(1).
Add a "bootstrap-register" target used specially by the bootstrap script to register packages that were installed by the bootstrap script.
Rename <phase>_COOKIE variables to _COOKIE.<phase>. These are private variables so there are no user-visible changes. This change just makes it a little easier to write for loops.
Flatten out recursive makes into a single re-invocation of make by introducing the concept of a "barrier". We separate the user-invokable targets into ones that must happen before the barrier, and ones that must happen after the barrier. The ones that happen after the barrier are run in a sub-make process. In this case, the targets that must be run after the barrier are from the "wrapper" step and beyond. We rewrite the various "flow" targets, e.g. wrapper, configure, build, etc., so that they of the right form to use the barrier target. This now completely removes the concept of PKG_PHASE from pkgsrc. It is replaced with the concept of "before" and "after" the barrier, and this state can be checked by testing for the existence of the barrier cookie file. Because we've removed most of the recursive makes, there is now nowhere to hook the PKG_ERROR_HANDLER.* commands, so remove them for now. As part of this commit, put back the logic that conditionalized the sources for the various cookie files. Because the sources are all "phony" targets, they were always run, regardless of whether or not the cookie file already existed. Now, if a cookie file exists, then that entire phase associated with that cookie file is skipped. Lastly, fix a thinko in configure/bsd.configure.mk where setting NO_CONFIGURE in a package Makefile would manage to skip the "wrapper" step altogether. Fix this by correctly noting "wrapper" and not "patch" as the preceding step to "configure".
Make the check-vulnerable target more self-sufficient, by moving some of the logic from fetch/fetch.mk into flavor/pkg/check.mk, so that check-vulnerable can be used as a source target. Make check-vulnerable a source target for every phase of the build workflow, which ensures that it is always run if the user starts a new phase from the command line. Fix the cookie-generation targets so that they don't append, only overwrite to the cookie file. This works around potential problems due to recursive makes. Move the cookie checks so that they surround the corresponding phase target. The presence of the cookie should now inform the make process to avoid doing any processing of phases that occur before the phase corresponding to the cookie.
Insert cookie checks into the "real-install" and "real-package" targets, which are invoked in recursive make calls for the "install" and "package" targets respectvely. These recursive make calls prevent the top-level make from seeing all of the targets and computing a full dependency graph, so it becomes possible for some targets to be invoked more than once. This change passes enough information along to the recursive make calls and ensures that the source targets for the real-* targets are only invoked once.
Restore missing ${SETENV} ${BUILD_ENV} that was dropped during the initial
package system flavors commit.
Introduce the capability to gather all the warnings and errors that
are generated for a target and output them all at once at the conclusion
of the target's invocation. The implementation is in bsd.pkg.error.mk,
which defines a macro target "error-check" that will print out any
non-empty warning and error files in ${WARNING_DIR} and ${ERROR_DIR}
and exit appropriately if there were errors.
Convert some targets that were just long sequences of ${ERROR_MSG} or
${WARNING_MSG} within a single shell statement to use the new delayed
error output via error-check.
Modify the compiler "fail" wrappers for C++ and Fortran to be less
verbose during invocation. Instead collect the warnings and only
print them at the end of the completed phase, e.g. after "configure"
and/or "build" completes.
If the install cookie file exists, then we don't need to do anything. Also, move the interactive check into a separate target as a prelude to removing the recursive make for invoking "real-install".
1.) Remove all instances of '@${WARNING_MSG} "Warning: ...' which resulted
in error message like 'WARNING: Warning: ...'.
2.) Replace "WARN_MSG" with "WARNING_MSG" which makes the "make package"
target work again for restricted packages like "acroread7".
Use PHASE_MSG, STEP_MSG, WARNING_MSG, and ERROR_MSG in place of ECHO_MSG in various places.
Allow overrides of the public targets, e.g. "install", "package", etc. if PKG_SKIP_REASON or PKG_FAIL_REASON is defined. This commit adds !target(...) guards around those target definitions to avoid "duplicate target definition" warnings.
First pass at implementing support for package system flavors other
than pkgsrc's current one. This is an important lead-up to any project
that redesigns the pkg_* tools in that it doesn't tie us to past design
(mis)choices. This commit mostly deals with rearranging code, although
there was a considerable amount of rewriting done in cases where I
thought the code was somewhat messy and was difficult to understand.
The design I chose for supporting multiple package system flavors is
that the various depends, install, package, etc. modules would define
default targets and variables that may be overridden in files from
pkgsrc/mk/flavor/${PKG_FLAVOR}. The default targets would do the
sensible thing of doing nothing, and pkgsrc infrastructure would rely
on the appropriate things to be defined in pkgsrc/mk/flavor to do the
real work. The pkgsrc/mk/flavor directory contains subdirectories
corresponding to each package system flavor that we support. Currently,
I only have "pkg" which represents the current pkgsrc-native package
flavor. I've separated out most of the code where we make assumptions
about the package system flavor, mostly either because we directly
use the pkg_* tools, or we make assumptions about the package meta-data
directory, or we directly manipulate the package meta-data files, and
placed it into pkgsrc/mk/flavor/pkg.
There are several new modules that have been refactored out of bsd.pkg.mk
as part of these changes: check, depends, install, package, and update.
Each of these modules has been slimmed down by rewriting them to avoid
some recursive make calls. I've also religiously documented which
targets are "public" and which are "private" so that users won't rely
on reaching into pkgsrc innards to call a private target.
The "depends" module is a complete overhaul of the way that we handle
dependencies. There is now a separate "depends" phase that occurs
before the "extract" phase where dependencies are installed. This
differs from the old way where dependencies were installed just before
extraction occurred. The reduce-depends.mk file is now replaced by
a script that is invoked only once during the depends phase and is
used to generate a cookie file that holds the full set of reduced
dependencies. It is now possible to type "make depends" in a package
directory and all missing dependencies will be installed.
Future work on this project include:
* Resolve the workflow design in anticipation of future work on
staged installations where "package" conceptually happens before
"install".
* Rewrite the buildlink3 framework to not assume the use of the
pkgsrc pkg_* tools.
* Rewrite the pkginstall framework to provide a standard pkg_*
tool to perform the actions, and allowing a purely declarative
file per package to describe what actions need to be taken at
install or deinstall time.
* Implement support for the SVR4 package flavor. This will be
proof that the appropriate abstractions are in place to allow
using a completely different set of package management tools.