Up to [cvs.NetBSD.org] / pkgsrc / mk
Request diff between arbitrary revisions
Keyword substitution: kv
Default branch: MAIN
mk: Add mktool support to wrapper. mktool is an optional external tool, available from pkgtools/mktool or "cargo install mktool", that provides significant performance improvements compared to the default wrapper target, while being 100% compatible. After installing, set TOOLS_PLATFORM.mktool=/path/to/mktool to enable it. Comparing the time taken to run "bmake wrapper" on a SmartOS host inside x11/kde-workspace4, the default wrapper target takes: real 1m42.699s user 0m27.931s sys 1m17.968s while mktool takes: real 0m9.090s user 0m4.571s sys 0m5.405s This is comparable to mk-buildlink-symlinks from pkgtools/mktools but does not require an additional dependency, so works against all packages, and also saves an extra package install for every package in a bulk build.
mk: Pass through all CROSSVARS as TARGET_* for tool depends.
Use this for cross-libtool-base.
Name it cross-libtool-base-${MACHINE_PLATFORM} instead of
cross-libtool-base-${MACHINE_ARCH}. MACHINE_GNU_PLATFORM isn't quite
enough, because it doesn't reflect OS versions (e.g., aarch64--netbsd
covers both NetBSD 9 and NetBSD 10).
No functional change intended in native builds -- everything here is
conditional on cross builds. The _DEPENDS_INSTALL_CMD shell fragment
is changed in native builds, but only to move around some of the
shell-based cross-build fragments (which could maybe be factored out
to save a bit of shell execution in native builds, but that's for
another day).
XXX MACHINE_PLATFORM doesn't cover all the other little exotic OS
knobs like LOWER_OPSYS_VERSUFFIX. Do we care?
Split native and cross LOCALBASE for cross-compilation.
This way, you can use an unprivileged prefix of ~/cross/pkg to
cross-compile packages that, when installed, will go in /usr/pkg.
In mk.conf:
- Set CROSS_LOCALBASE to determine the installation prefix of
packages that you cross-compile.
- Set LOCALBASE to determine the installation prefix of native
packages needed for cross compilation.
Both default to /usr/pkg. Note: CROSS_LOCALBASE defaults to /usr/pkg
even if you set LOCALBASE. Of course, if you're not cross-compiling,
this has no effect.
When cross-compiling a package:
- LOCALBASE is where cross-compiled packages will be installed.
(`make install' will put it in ${CROSS_DESTDIR}${LOCALBASE}, of
course, but when you later use pkg_add to install the binary
package on the system you're compiling it for, it will go in
${LOCALBASE}. And you can still set PREFIX for an individual
package, but LOCALBASE refers to where cross-compiled packages are
generally installed, other than the specific ones that override
PREFIX.)
- TOOLBASE is where tool dependencies are installed and available to
be executed at build-time.
For example, if you have TOOL_DEPENDS on a program that is
installed in (say) foo/bin/xyz which is not in PATH so you have to
refer ot its full pathname, you can execute ${TOOLBASE}/foo/bin/xyz
instead of ${LOCALBASE}/foo/bin/xyz.
I chose this split, rather than the other way around, because I
expect that references that are baked into installed packages to be
more common than references that are executed at build-time, but I
didn't do an extensive study of the matter. I chose the name
TOOLBASE because it's where TOOL_DEPENDS go.
bsd.pkg.use.mk: TOOL_DEPENDS, not BUILD_DEPENDS, for libtool-fortran. But we need a cross-libtool-fortran for this to actually work in cross compilation. No functional change intended for native compilation, since BUILD_DEPENDS and TOOL_DEPENDS have the same semantics there.
mk: Add PKGSRC_USE_MKTOOLS support. When enabled, pull in the pkgtools/mktools package to use C-based tools for certain parts of the pkgsrc mk infrastructure. Default is off for now while any portability issues are shaken out, but it has been tested successfully on at least illumos, macOS, and NetBSD. The first tool is mk-buildlink-symlinks which creates the buildlink3 symlinks as part of the "wrapper" phase. This significantly improves performance, especially with packages that have a lot of buildlink3 dependencies. For example, the time for "bmake wrapper" in x11/kde-workspace4 goes from: real 3:20.696394375 user 50.553556463 sys 2:23.883823896 to: real 19.873100598 user 8.141441461 sys 11.740602820 It's expected that other tools will be added over time to speed up other parts of the infrastructure.
mk: Stop defaulting to -Werror with BSD makefiles outside the NetBSD source tree
mk: When using imake, pass LDFLAGS in the environment as LOCAL_LDFLAGS. This allows RELRO to be used when building imake packages. XXX: I'm not certain if this is the right place, but this already seems to be cargo-culted in individual package Makefiles...
mk/bsd.pkg.use.mk: add more keywords for "bmake help" The added empty line separates this help topic from the header comment of this file, since USE_BSD_MAKEFILE does not add any dependencies and therefore doesn't fit that description.
Sync cross-libtool-base with libtool-base. Eliminate CROSSBASE. cross-libtool-base now installs into $PREFIX/cross-$TARGET_ARCH unconditionally.
Remove traces of crypto restrictions. ok for idea riastradh
Fix path to cross-libtool-base in TOOL_DEPENDS+= for crossbuilding
Coorect version of libtool to current one.
Bump LIBTOOL_REQD, newer version needed for cwrappers. PR 51826.
Handle optional ctfmerge/ctfconvert presence on NetBSD by explicitly passing down NOCTF when they don't exist.
Get rid of INSTALLATION_PREFIX.
Remove support for USE_CROSSBASE.
Remove USE_X11BASE and X11PREFIX.
Drop _LIBTOOL and _SHLIBTOOL and make LIBTOOL and SHLIBTOOL point to the wrapped version by full path. This fixes some cases where the wrappers have been bypassed. lang/lua52 triggered the investigation as it failed to link against libreadline, which should have been translated to libedit. Tested by jperkin and myself with full bulk builds.
Remove pkg_views support, second part: infrastructure.
Remove mk glue for devel/nbitools.
Only add HAVE_LLVM / HAVE_GCC if they are already set.
Pass down HAVE_LLVM and HAVE_GCC.
check defined before using USE_LANGUAGES as condition.
Split BUILD_DEPENDS into TOOL_DEPENDS and BUILD_DEPENDS in mk/. Build depends are target packages that are needed at build-time for, e.g., static libraries to link against, header files to include, &c. Tool depends are native packages that are needed at build-time for, e.g., compilers/linkers/&c. to run. ok agc
Split Fortran support from devel/libtool-base into a separate package devel/libtool-fortran. If USE_LIBTOOL is present and Fortran is in USE_LANGUAGES, include the new package.
Drop imake-check logic from X11 components' builtin.mk. Always use xorg-cf-files and imake from pkgsrc, replacing xpkgwedge. Always install man pages, not cat pages when using imake. Unify the various imake PLIST variables in preparation for dropping. Adjust xbattbar for the new expectations.
Fix libtool under OS X: require a fixed version of f2c, and thus bump PKGREVISION of libtool to 3 and require it as a dependency.
Force libtool 2.2 on all platforms.
Remove USE_DIRS infrastructure support and support packages. See http://mail-index.netbsd.org/pkgsrc-changes/2009/07/22/msg026834.html.
If we have USE_BSD_MAKEFILE, pass down INSTALL=${TOOLS_INSTALL:Q} in the
environment.
Otherwise BSD style packages will use the install(1) which
bootstrap-mk-files' sys.mk picked, rather than the one from
TOOLS_PLATFORM.install (which may be specified by the user in mk.conf).
Get rid of USE_TOOLS+=nroff in the USE_BSD_MAKEFILE case. Just because a package has BSD style Makefile doesn't mean it has manpages. A sweep of packages with USE_BSD_MAKEFILE=yes is forthcoming; USE_TOOLS+=groff nroff will be added where appropriate.
Documented USE_IMAKE.
Made the .include lines simpler.
Remove USE_MTREE support. Keep one copy (the NetBSD version) for the purpose of print-PLIST in plist/common-dirs.mtree. Discussed with wiz@, no objections on tech-pkg@.
Make MTREE_FILE and MTREE_ARGS internal by prefixing them with '_'. They are not supposed to be set by the user.
Replaced the deprecated PKG_SKIP_REASON with PKG_FAIL_REASON.
Reverted the change between 1.37 and 1.38. MANPATH cannot be overridden so easily to make it right for imake Packages. Noticed by Dan.
Added the variable USE_BSD_MAKEFILE.
Fixed PKGMANDIR for all packages using Imake.
For modular Xorg disable xpkgwedge (will be made a hard error later).
Don't add ${X11BASE}/bin to PATH, don't include mk/x11.buildlink3.mk
when USE_X11BASE is set and don't use BUILDLINK_X11_DIR and related
magic.
OKed by wiz@
No need to define LIBTOOL_OVERRIDE explicitly since libtool-override.mk
will search for libtool scripts up to ${OVERRIDE_DIRDEPTH} below
${WRKSRC} already.
Teach the tools framework about the following new tools:
itools, intltool, diff3, sdiff, msgmerge
* Adding USE_TOOLS+=itools to a package Makefile will cause the
tool-directory versions of imake, makedepend, mkdirhier and xmkmf
to point to the ones from the devel/nbitools package.
This change will remove the need for nbitools/buildlink3.mk, which
currently does a bit of hackery to force the "right" imake tools to
be used by packages that need it.
* Adding USE_TOOLS+=intltool to a package Makefile will cause the
local versions of intltool-* inside ${WRKSRC} to be replaced by
copies from the textproc/intltool package. If "intltool" is not
specified as a tool, then we create "broken" intltool-* tools in
the tools directory to help highlight hidden dependencies on the
intltool package.
In addition, modify the tools framework so that if "perl" is not
specified as a tool, then we create a "broken" perl tool in the
tools directory for the same reason as for "intltool".
These two changes together will remove the need for
intltools/buildlink3.mk and should also catch all cases where the
sources' intltools may have been silently used because perl was
found on the system.
* Adding USE_TOOLS+=diff3, USE_TOOLS+=sdiff, or USE_TOOLS+=msgmerge
to a package Makefile will cause the corresponding tool to be pulled
into the tools directory.
These are convenience tools to help simplify dependencies for some
packages.
EOL the tv-derecurse branch; it has become too outdated. The last live revision can be fetched at the tv-derecurse-eol tag.
First pass at implementing support for package system flavors other
than pkgsrc's current one. This is an important lead-up to any project
that redesigns the pkg_* tools in that it doesn't tie us to past design
(mis)choices. This commit mostly deals with rearranging code, although
there was a considerable amount of rewriting done in cases where I
thought the code was somewhat messy and was difficult to understand.
The design I chose for supporting multiple package system flavors is
that the various depends, install, package, etc. modules would define
default targets and variables that may be overridden in files from
pkgsrc/mk/flavor/${PKG_FLAVOR}. The default targets would do the
sensible thing of doing nothing, and pkgsrc infrastructure would rely
on the appropriate things to be defined in pkgsrc/mk/flavor to do the
real work. The pkgsrc/mk/flavor directory contains subdirectories
corresponding to each package system flavor that we support. Currently,
I only have "pkg" which represents the current pkgsrc-native package
flavor. I've separated out most of the code where we make assumptions
about the package system flavor, mostly either because we directly
use the pkg_* tools, or we make assumptions about the package meta-data
directory, or we directly manipulate the package meta-data files, and
placed it into pkgsrc/mk/flavor/pkg.
There are several new modules that have been refactored out of bsd.pkg.mk
as part of these changes: check, depends, install, package, and update.
Each of these modules has been slimmed down by rewriting them to avoid
some recursive make calls. I've also religiously documented which
targets are "public" and which are "private" so that users won't rely
on reaching into pkgsrc innards to call a private target.
The "depends" module is a complete overhaul of the way that we handle
dependencies. There is now a separate "depends" phase that occurs
before the "extract" phase where dependencies are installed. This
differs from the old way where dependencies were installed just before
extraction occurred. The reduce-depends.mk file is now replaced by
a script that is invoked only once during the depends phase and is
used to generate a cookie file that holds the full set of reduced
dependencies. It is now possible to type "make depends" in a package
directory and all missing dependencies will be installed.
Future work on this project include:
* Resolve the workflow design in anticipation of future work on
staged installations where "package" conceptually happens before
"install".
* Rewrite the buildlink3 framework to not assume the use of the
pkgsrc pkg_* tools.
* Rewrite the pkginstall framework to provide a standard pkg_*
tool to perform the actions, and allowing a purely declarative
file per package to describe what actions need to be taken at
install or deinstall time.
* Implement support for the SVR4 package flavor. This will be
proof that the appropriate abstractions are in place to allow
using a completely different set of package management tools.
No package is using USE_RMAN any longer, remove it.
BUILD_USE_MSGFMT and USE_MSGFMT_PLURALS are obsolete. Replace with USE_TOOLS+=msgfmt.
Overhaul the way packages can ask for "msgfmt". If a package needs msgfmt, then it should set the following in the package Makefile: USE_TOOLS+= msgfmt To deal with message files that use the "msgid_plural" statement, which isn't supported in NetBSD<=3.x and also in gettext<=0.10.35, we determine if the built-in "msgfmt" is sufficiently new enough to understand "msgid_plural". If it isn't, then we use the msgfmt.sh script to transform the msgid_plural statements to an equivalent construct that's understood by older msgfmt tools. The msgfmt.sh script is a straightforward translation of the original perl script msgfmt.pl script by Julio M. Merino Vidal into shell and awk, which are more lightweight dependencies than perl. We remove the USE_MSGFMT_PLURALS bits in gettext-lib/builtin.mk as they are made obsolete by the new code in mk/tools/msgfmt.mk. BUILD_USE_MSGFMT is still supported but will be removed in a separate commit.
BUILD_DEPENDS on gettext-tools not gettext for BUILD_USES_MSGFMT.
Reimplement the info-file handling so that we use an +INFO_FILES scriptlet to manage the info-file registration. The new scriptlet's template is install/info-files. Remove obsolete texinfo.mk and install/install-info. No changes to package Makefiles are necessary -- the re-implementation is internal to pkgsrc infrastructure.
* Teach the tools framework how to supply the pkgsrc version of
makeinfo if no native makeinfo executable exists. Honor TEXINFO_REQD
when determining whether the native makeinfo can be used.
* Remove USE_MAKEINFO and replace it with USE_TOOLS+=makeinfo.
* Get rid of all the "split" argument deduction for makeinfo since
the PLIST module already handles varying numbers of split info files
correctly.
NOTE: Platforms that have "makeinfo" in the base system should check
that the makeinfo entries of pkgsrc/mk/tools.${OPSYS}.mk are
correct.
Require xpkgwedge 1.15.
Don't ever depend on xpkgwedge if we're building xpkgwedge.
Applied all quoting fixes found by "pkglint --autofix".
Add IMAKE_GAMEMAN_SUFFIX and IMAKE_GAMEMAN_DIR to allow PLISTs to pick up the correct locations. XXX Default values are guessed, x11/xsnow can be used for testing
Bump LIBTOOL_REQD to 1.5.18nb5. This pulls in the -avoid-version fix for some platforms, which includes all non-ELFs and many ELF-like platforms (that still use a.out naming conventions). Since a branch is coming, bump the version in a blanket rather than per platform.
Tools that are marked with ":pkgsrc" are needed very early on, probably too early for pkgsrc to adequately cope. In this case, imake-check.mk was marking "imake" as a tool that was used to perform some tests. This was causing xpkgwedge to be unnecessarily marked as a build dependency since using imake in pkgsrc pretty much requires xpkgwedge. However, in the case where we are running the "imake checks", we don't need xpkgwedge around. Solve this issue by marking all the tools in imake-check.mk with ":pkgsrc", and modify the xpkgwedge dependency test to not care about "imake:pkgsrc".
Clean up some clutter in pkgsrc/mk: move the ${OPSYS}.*.dist mtree
files into mk/platform, where they now live with the ${OPSYS}.mk files.
Require xpkgwedge>=1.14 so that the correct make is always invoked on Solaris and Linux, regardless of whether we use the same make to build the software or not.
We need xpkgwedge whenever we use imake.
Require xpkgwedge>=1.13 when used as a build dependency so that the correct make(1) program is invoked by pkgxmkmf.
We only need to set USE_X11BASE, run install.man, and override the compiler (CC/CXX) if we're actually using imake to generate Makefiles for building and installing software. This fixes errors in various KDE-3.x packages that use imake for other purposes.
Massive cleanup of buildlink3.mk and builtin.mk files in pkgsrc.
Several changes are involved since they are all interrelated. These
changes affect about 1000 files.
The first major change is rewriting bsd.builtin.mk as well as all of
the builtin.mk files to follow the new example in bsd.builtin.mk.
The loop to include all of the builtin.mk files needed by the package
is moved from bsd.builtin.mk and into bsd.buildlink3.mk. bsd.builtin.mk
is now included by each of the individual builtin.mk files and provides
some common logic for all of the builtin.mk files. Currently, this
includes the computation for whether the native or pkgsrc version of
the package is preferred. This causes USE_BUILTIN.* to be correctly
set when one builtin.mk file includes another.
The second major change is teach the builtin.mk files to consider
files under ${LOCALBASE} to be from pkgsrc-controlled packages. Most
of the builtin.mk files test for the presence of built-in software by
checking for the existence of certain files, e.g. <pthread.h>, and we
now assume that if that file is under ${LOCALBASE}, then it must be
from pkgsrc. This modification is a nod toward LOCALBASE=/usr. The
exceptions to this new check are the X11 distribution packages, which
are handled specially as noted below.
The third major change is providing builtin.mk and version.mk files
for each of the X11 distribution packages in pkgsrc. The builtin.mk
file can detect whether the native X11 distribution is the same as
the one provided by pkgsrc, and the version.mk file computes the
version of the X11 distribution package, whether it's built-in or not.
The fourth major change is that the buildlink3.mk files for X11 packages
that install parts which are part of X11 distribution packages, e.g.
Xpm, Xcursor, etc., now use imake to query the X11 distribution for
whether the software is already provided by the X11 distribution.
This is more accurate than grepping for a symbol name in the imake
config files. Using imake required sprinkling various builtin-imake.mk
helper files into pkgsrc directories. These files are used as input
to imake since imake can't use stdin for that purpose.
The fifth major change is in how packages note that they use X11.
Instead of setting USE_X11, package Makefiles should now include
x11.buildlink3.mk instead. This causes the X11 package buildlink3
and builtin logic to be executed at the correct place for buildlink3.mk
and builtin.mk files that previously set USE_X11, and fixes packages
that relied on buildlink3.mk files to implicitly note that X11 is
needed. Package buildlink3.mk should also include x11.buildlink3.mk
when linking against the package libraries requires also linking
against the X11 libraries. Where it was obvious, redundant inclusions
of x11.buildlink3.mk have been removed.
Remove the old tools framework and references to _USE_NEW_TOOLS.
Goal: Remove USE_PERL5 from pkgsrc. Plan: (1) Change USE_PERL5=build into USE_TOOLS+=perl. (2) Change all other USE_PERL5 into including perl5/buildlink3.mk. Possibly, for packages that don't actually build anything with perl, but merely require it for the perl interpreter, we can instead do: USE_TOOLS+= perl TOOLS_DEPMETHOD.perl= DEPENDS but this is more verbose than simply including the perl5/buildlink3.mk file. Move the PERL5_REQD computation into a lang/perl5/version.mk file, and only do the USE_PERL5 logic in bsd.pkg.use.mk if we're not using the new tools framework. This consolidates all of the perl-handling into two places -- lang/perl5 and mk/tools/perl.mk.
Since there may have been merge errors in libtool-base 1.5.18, force LIBTOOL_REQD to 1.5.18nb1.
If USE_TOOLS has "perl" then also define PERL5_PKGSRCDIR for use by mk/tools/perl.mk. This fixes broken dependencies on perl where the path to the package wasn't defined.
Make USE_TOOLS+=imake imply USE_X11BASE=yes, along with the other variable definitions that apply when USE_IMAKE is defined.
sync [all but bsd.pkg.mk] with HEAD
Teach the new tools framework more about perl. We include perl5/vars.mk in mk/tools/perl.mk since many packages expect to be able to use the variables defined in vars.mk, but those variables can only be defined if PERL5 is correctly defined, and that is only true after it is set here.
Don't check for whether variables are defined or not before adding
them to PLIST_SUBST. They may simply be defined later on in the
Makefile processing. Instead check whether USE_PERL5 was specified
to determine whether to pass along PERL5_{SITEARCH,SITELIB,ARCHLIB}
to PLIST_SUBST.
Sense of a USE_BUILDLINK3 test was reversed when USE_BUILDLINK3 was switched on always, resulting in instances of X11R6/lib libraries being linked in to binaries along with pkgsrc versions of the same. Fix.
USE_BUILDLINK3 is no longer optional (and cannot be turned off). Per mail to tech-pkg: ===== * USE_BUILDLINK3=YES will be unconditional. (In fact, USE_BUILDLINK3 will be ignored altogether by mk/; but see below.) * NO_BUILDLINK and NO_WRAPPER will be ignored by mk/. If a build happens, these phases will happen. * The existing NO_BUILD will imply the previous NO_BUILDLINK and NO_WRAPPER. If no build happens, those phases are not needed. * NO_TOOLS will be ignored by mk/. The tools phase, which provides much more than just the C compiler, will always happen regardless of package. This will make metapackage builds only slightly slower, in trade for far less user error.
Bump LIBTOOL_REQD to 1.5.14; this unifies all the support again and fixes a good number of bugs.
sync with HEAD
* Teach devel/p5-Module-Build to write .packlists just like MakeMaker.pm does. This allows us to use dynamic PLISTs for Perl modules that are built using Module::Build. Bump the PKGREVISION of p5-Module-Build to 1. * Drop the use of PERL5_USES_MODULE_BUILD and introduce a new variable PERL5_MODULE_TYPE that is either "MakeMaker" or "Module::Build" that names the framework used to build/install the module. * Split out the variables set in perl5/buildlink3.mk that are also used by perl5/module.mk into a new file perl5/vars.mk. Move some PERL5_* variable definitions from pkgsrc/mk/bsd.pkg.use.mk into perl5/vars.mk. This just centralizes the common PERL5_* definitions into a single file location. * Convert the known packages that use Module::Build to set PERL5_MODULE_TYPE and PERL5_PACKLIST: devel/p5-Class-Container devel/p5-Exception-Class devel/p5-Log-Dispatch devel/p5-Array-Compare textproc/p5-Pod-Coverage www/p5-Apache-Session-Wrapper www/p5-MasonX-Request-WithApacheSession
Migrate the big block of USE_PERL5 and PERL5_REQD logic to bsd.pkg.use.mk.
Migrate several USE_* logic blocks, previously sprinkled liberally throughout bsd.pkg.mk, to labelled blocks in bsd.pkg.use.mk.
Merge down some more visual cleanup from tv-derecurse to reduce divergence. Part of this, introduce bsd.pkg.use.mk, where the Special Logic invoked by various pkgsrc-wide USE_* variables will be migrated/consolidated.
sync with HEAD
sync with HEAD
sync with HEAD
sync with HEAD
sync with HEAD
sync with HEAD
Merge from HEAD.
So that I don't lose all this work, stash the work-in-progress derecursion cleanup onto the tv-derecurse branch (only branched in pkgsrc/mk).
file bsd.pkg.use.mk was initially added on branch tv-derecurse.