Return to HOWTO-pbulk CVS log

Up to [cvs.NetBSD.org] / pkgsrc / doc

Add a note about sharing sandboxes between master and slave based on
discussion with Havard. Add a reference to the bulkbuild-restart script.

Step One: Suggested File System Layout

For the DragonFly builds I'm using:
- /bulklog for the bulk meta data and log files of each package
- /scratch as WRKOBJDIR
- /distfiles as DISTDIR
- /packages as PACKAGES
- separate physical file systems for /var and /usr/pkg
All file systems but those are read-only, /tmp and /dev need platform
specific handling either as tmpfs or link. ptyfs helps for /dev.

For parallel builds, /bulklog, /packages and /usr/pkgsrc are supposed to
be shared (null mounts, NFS). /usr/pkg and /var/db/pkg needs to be
per-node and /scratch should be per-node.

The defaults scripts will remove /usr/pkg and /var/db/pkg on the master
node as well. When using chroot sandboxes for the build, the master can
share a sandbox with one of the build instances.

Step Two: Preparing pbulk prefix

Install pkgtools/pbulk and possibly helper programs like screen into
LOCALBASE=/usr/pkg_bulk, PKG_DBDIR=/usr/pkg_bulk/.pkgdb. A full
bootstrap is the recommended approach. Copy /usr/pkg_bulk to the client
nodes for parallel builds.

Step Three: Configure pbulk

(a)
Setup general build parameters in mk.conf. Below is an example
fragment to add.

(b) [not for NetBSD]
Build a binary bootstrap kit and override the included mk.conf with the
file created in (a). Compress the tarball with gzip.

(c)
Edit /usr/pkg_bulk/etc/pbulk.conf:
- For NetBSD:
  - pkg_install_prefix should be /usr.
  - bootstrapkit should be empty.
- For non-NetBSD:
  - pkg_install_prefix should /usr/pkg_bulk.
  - make must be changed to /usr/pkg/bin/bmake.
  - bootstrapkit should be changed to the tarball in (b). Both master
    and client nodes access this, so keep it in a shared location.

- master_mode=yes needs a proper list of client nodes.
- base_url goes into the report mail, the rsync targets are used
  for uploads.
- for limited bulk builds, set limited_list to a file with one
  location per line

Step Four: Running pbulk

/usr/pkg_bulk/bin/bulkbuild in screen runs the full build.

/usr/pkg_bulk/libexec/pbulk/{pre-build,scan,build,report,upload} are the
individual build phases. They depend on the execution of the phase
before. Exceptions are report and upload which both depend on build.

/usr/pkg_bulk/bin/bulkbuild-restart can be used to resume an aborted
build. If it failed after the actual build phase, report and upload should
be called individually though.

--- sample mk.conf fragment ---

WRKOBJDIR = /scratch
PKGSRCDIR = /usr/pkgsrc
DISTDIR = /distfiles
PACKAGES = /packages

FAILOVER_FETCH=	yes

X11_TYPE=	modular

_ACCEPTABLE=			yes
ALLOW_VULNERABLE_PACKAGES=	yes

PKG_DEVELOPER?= yes