Return to HOWTO-pbulk CVS log

Up to [cvs.NetBSD.org] / pkgsrc / doc

Annotation of pkgsrc/doc/HOWTO-pbulk, Revision 1.7

1.3       wiz         1: Step One: Suggested File System Layout
1.1       joerg       2:
                      3: For the DragonFly builds I'm using:
                      4: - /bulklog for the bulk meta data and log files of each package
                      5: - /scratch as WRKOBJDIR
                      6: - /distfiles as DISTDIR
                      7: - /packages as PACKAGES
1.3       wiz         8: - separate physical file systems for /var and /usr/pkg
                      9: All file systems but those are read-only, /tmp and /dev need platform
1.1       joerg      10: specific handling either as tmpfs or link. ptyfs helps for /dev.
                     11:
                     12: For parallel builds, /bulklog, /packages and /usr/pkgsrc are supposed to
                     13: be shared (null mounts, NFS). /usr/pkg and /var/db/pkg needs to be
                     14: per-node and /scratch should be per-node.
                     15:
1.4       joerg      16: The defaults scripts will remove /usr/pkg and /var/db/pkg on the master
                     17: node as well. When using chroot sandboxes for the build, the master can
                     18: share a sandbox with one of the build instances.
                     19:
1.1       joerg      20: Step Two: Preparing pbulk prefix
                     21:
                     22: Install pkgtools/pbulk and possibly helper programs like screen into
                     23: LOCALBASE=/usr/pkg_bulk, PKG_DBDIR=/usr/pkg_bulk/.pkgdb. A full
1.3       wiz        24: bootstrap is the recommended approach. Copy /usr/pkg_bulk to the client
1.1       joerg      25: nodes for parallel builds.
                     26:
1.5       dmcmahil   27: The initial bootstrap and install may be performed with the following commands:
                     28:
                     29:     cd /usr/pkgsrc/bootstrap
                     30:     ./bootstrap --prefix=/usr/pkg_bulk --pkgdbdir=/usr/pkg_bulk/.pkgdb
                     31:
                     32: If you have already performed a bootstrap into /usr/pkg_bulk then you may need
                     33: to remove that directory (rm -fr /usr/pkg_bulk) prior to starting the new
                     34: bootstrap.
                     35:
                     36: pbulk may be installed into /usr/pkg_bulk with the following:
                     37:
                     38: cd /usr/pkgsrc/pkgtools/pbulk
                     39: env PATH=/usr/pkg_bulk/bin:/usr/pkg_bulk/sbin:${PATH} bmake package
                     40:
                     41: The mk.conf file in /usr/pkg_bulk/etc/mk.conf should already have the
                     42: correct LOCALBASE and PKG_DBDIR settings.
                     43:
1.1       joerg      44: Step Three: Configure pbulk
                     45:
                     46: (a)
                     47: Setup general build parameters in mk.conf. Below is an example
1.5       dmcmahil   48: fragment to add.  On NetBSD this is typically /etc/mk.conf.  For
                     49: non-NetBSD systems we will create a mk.conf in a temporary location,
                     50: /tmp/mk.conf for example, and include it in the bootstrap that will
                     51: be built in step (b).
1.1       joerg      52:
1.3       wiz        53: (b) [not for NetBSD]
1.1       joerg      54: Build a binary bootstrap kit and override the included mk.conf with the
1.5       dmcmahil   55: file created in (a). Compress the tarball with gzip.  This is a new
                     56: bootstrap that is different from the one created in Step Two above.
                     57:
                     58: This bootstrap kit is created by the following:
                     59:
                     60:
                     61:     cd /usr/pkgsrc/bootstrap
                     62:     ./bootstrap --mk-fragment /tmp/mk.conf \
1.6       joerg      63:         --gzip-binary-kit bootstrap_kit.tar.gz
1.5       dmcmahil   64:
                     65: Note that we have specified the make file fragment, /tmp/mk.conf, created
                     66: in step (a) above.
1.1       joerg      67:
                     68: (c)
                     69: Edit /usr/pkg_bulk/etc/pbulk.conf:
                     70: - For NetBSD:
                     71:   - bootstrapkit should be empty.
1.5       dmcmahil   72:   - make must be changed to /usr/bin/make.
                     73:
1.1       joerg      74: - For non-NetBSD:
                     75:   - bootstrapkit should be changed to the tarball in (b). Both master
                     76:     and client nodes access this, so keep it in a shared location.
                     77:
                     78: - master_mode=yes needs a proper list of client nodes.
                     79: - base_url goes into the report mail, the rsync targets are used
                     80:   for uploads.
                     81: - for limited bulk builds, set limited_list to a file with one
1.5       dmcmahil   82:   location per line ("x11/gtk2" for example).  Packages that are required
                     83:   as dependencies will be added automatically.
1.1       joerg      84:
                     85: Step Four: Running pbulk
                     86:
1.5       dmcmahil   87: /usr/pkg_bulk/bin/bulkbuild runs the full build.  You may wish to run
                     88: this inside of screen (misc/screen) since a full bulk build may take
                     89: a very long time.
1.1       joerg      90:
                     91: /usr/pkg_bulk/libexec/pbulk/{pre-build,scan,build,report,upload} are the
                     92: individual build phases. They depend on the execution of the phase
                     93: before. Exceptions are report and upload which both depend on build.
                     94:
1.4       joerg      95: /usr/pkg_bulk/bin/bulkbuild-restart can be used to resume an aborted
                     96: build. If it failed after the actual build phase, report and upload should
                     97: be called individually though.
                     98:
1.1       joerg      99: --- sample mk.conf fragment ---
                    100:
                    101: WRKOBJDIR = /scratch
                    102: PKGSRCDIR = /usr/pkgsrc
                    103: DISTDIR = /distfiles
                    104: PACKAGES = /packages
                    105:
                    106: FAILOVER_FETCH=        yes
                    107:
                    108: X11_TYPE=      modular
                    109:
1.7     ! joerg     110: SKIP_LICENSE_CHECK=                    yes
1.1       joerg     111: ALLOW_VULNERABLE_PACKAGES=     yes
                    112:
                    113: PKG_DEVELOPER?= yes