Annotation of htdocs/docs/pkgsrc/bulk.html, revision 1.2

1.1       rillig      1: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
                      2: <html>
                      3: <head>
                      4: <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
                      5: <title>Chapter 7. Creating binary packages for everything in pkgsrc (bulk builds)</title>
                      6: <link rel="stylesheet" href="/global.css" type="text/css">
                      7: <meta name="generator" content="DocBook XSL Stylesheets VX.X.X">
                      8: <link rel="start" href="index.html" title="The pkgsrc guide">
                      9: <link rel="up" href="users-guide.html" title="Part I. The pkgsrc user's guide">
                     10: <link rel="prev" href="binary.html" title="Chapter 6. Creating binary packages">
                     11: <link rel="next" href="files.html" title="Chapter 8. Directory layout of the installed files">
                     12: </head>
                     13: <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
                     14: <div class="navheader">
                     15: <table width="100%" summary="Navigation header">
                     16: <tr><th colspan="3" align="center">Chapter 7. Creating binary packages for everything in pkgsrc (bulk
                     17: builds)</th></tr>
                     18: <tr>
                     19: <td width="20%" align="left">
                     20: <a accesskey="p" href="binary.html">Prev</a> </td>
                     21: <th width="60%" align="center">Part I. The pkgsrc user's guide</th>
                     22: <td width="20%" align="right"> <a accesskey="n" href="files.html">Next</a>
                     23: </td>
                     24: </tr>
                     25: </table>
                     26: <hr>
                     27: </div>
                     28: <div class="chapter" lang="en">
                     29: <div class="titlepage"><div><div><h2 class="title">
                     30: <a name="bulk"></a>Chapter 7. Creating binary packages for everything in pkgsrc (bulk
                     31: builds)</h2></div></div></div>
                     32: <div class="toc">
                     33: <p><b>Table of Contents</b></p>
                     34: <dl>
                     35: <dt><span class="sect1"><a href="bulk.html#bulk.pre">7.1. Think first, build later</a></span></dt>
                     36: <dt><span class="sect1"><a href="bulk.html#bulk.req">7.2. Requirements of a bulk build</a></span></dt>
                     37: <dt><span class="sect1"><a href="bulk.html#bulk.old">7.3. Running an old-style bulk build</a></span></dt>
                     38: <dd><dl>
                     39: <dt><span class="sect2"><a href="bulk.html#binary.configuration">7.3.1. Configuration</a></span></dt>
                     40: <dt><span class="sect2"><a href="bulk.html#other-environmental-considerations">7.3.2. Other environmental considerations</a></span></dt>
                     41: <dt><span class="sect2"><a href="bulk.html#operation">7.3.3. Operation</a></span></dt>
                     42: <dt><span class="sect2"><a href="bulk.html#what-it-does">7.3.4. What it does</a></span></dt>
                     43: <dt><span class="sect2"><a href="bulk.html#disk-space-requirements">7.3.5. Disk space requirements</a></span></dt>
                     44: <dt><span class="sect2"><a href="bulk.html#setting-up-a-sandbox">7.3.6. Setting up a sandbox for chrooted builds</a></span></dt>
                     45: <dt><span class="sect2"><a href="bulk.html#building-a-partial-set">7.3.7. Building a partial set of packages</a></span></dt>
                     46: <dt><span class="sect2"><a href="bulk.html#bulk-upload">7.3.8. Uploading results of a bulk build</a></span></dt>
                     47: </dl></dd>
                     48: <dt><span class="sect1"><a href="bulk.html#bulk.pbulk">7.4. Running a pbulk-style bulk build</a></span></dt>
                     49: <dd><dl><dt><span class="sect2"><a href="bulk.html#bulk.pbulk.conf">7.4.1. Configuration</a></span></dt></dl></dd>
                     50: <dt><span class="sect1"><a href="bulk.html#creating-cdroms">7.5. Creating a multiple CD-ROM packages collection</a></span></dt>
                     51: <dd><dl><dt><span class="sect2"><a href="bulk.html#cdpack-example">7.5.1. Example of cdpack</a></span></dt></dl></dd>
                     52: </dl>
                     53: </div>
                     54: <p>When you have multiple machines that should run the same packages,
                     55: it is wasted time if they all build their packages themselves from
                     56: source. There are two ways of getting a set of binary packages: The old
                     57: bulk build system, or the new (as of 2007) parallel bulk build (pbulk)
                     58: system. This chapter describes how to set them up so that the packages
                     59: are most likely to be usable later.</p>
                     60: <div class="sect1" lang="en">
                     61: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
                     62: <a name="bulk.pre"></a>7.1. Think first, build later</h2></div></div></div>
                     63: <p>Since a bulk build takes several days or even weeks to finish, you
                     64: should think about the setup before you start everything. Pay attention
                     65: to at least the following points:</p>
                     66: <div class="itemizedlist"><ul type="disc">
                     67: <li>
                     68: <p>If you want to upload the binary packages to
                     69: ftp.NetBSD.org, make sure the setup complies to the requirements for binary
                     70: packages:</p>
                     71: <div class="itemizedlist"><ul type="circle">
                     72: <li><p>To end up on ftp.NetBSD.org, the packages must be built
                     73: by a NetBSD developer on a trusted machine (that is, where you and only
                     74: you have root access).</p></li>
                     75: <li><p>Packages on ftp.NetBSD.org should only be created from
                     76: the stable branches (like 2007Q1), so that users browsing the available
                     77: collections can see at a glance how old the packages
                     78: are.</p></li>
                     79: <li><p>The packages must be built as root, since some packages
                     80: require set-uid binaries at runtime, and creating those packages as
                     81: unprivileged user doesn't work well at the moment.</p></li>
                     82: </ul></div>
                     83: </li>
                     84: <li><p>Make sure that the bulk build cannot break anything in
                     85: your system. Most bulk builds run as root, so they should be run at least
                     86: in a chroot environment or something even more restrictive, depending on
                     87: what the operating system provides. There have been numerous cases where
                     88: certain packages tried to install files outside the
                     89: <code class="filename">LOCALBASE</code> or wanted to edit some files in
                     90: <code class="filename">/etc</code>. Furthermore, the bulk builds install and
                     91: deinstall packages in <code class="filename">/usr/pkg</code> (or whatever
                     92: <code class="filename">LOCALBASE</code> is) during their operation, so be sure
                     93: that you don't need any package during the build.</p></li>
                     94: </ul></div>
                     95: </div>
                     96: <div class="sect1" lang="en">
                     97: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
                     98: <a name="bulk.req"></a>7.2. Requirements of a bulk build</h2></div></div></div>
                     99: <p>A complete bulk build requires lots of disk space. Some of the
                    100: disk space can be read-only, some other must be writable. Some can be on
                    101: remote filesystems (such as NFS) and some should be local. Some can be
                    102: temporary filesystems, others must survive a sudden reboot.</p>
                    103: <div class="itemizedlist"><ul type="disc">
                    104: <li><p>10 GB for the distfiles (read-write, remote, temporary)</p></li>
                    105: <li><p>10 GB for the binary packages (read-write, remote, permanent)</p></li>
                    106: <li><p>400 MB for the pkgsrc tree (read-only, remote, permanent)</p></li>
                    107: <li><p>5 GB for <code class="filename">LOCALBASE</code> (read-write, local, temporary for pbulk, permanent for old-bulk)</p></li>
                    108: <li><p>5 GB for the log files (read-write, remote, permanent)</p></li>
                    109: <li><p>5 GB for temporary files (read-write, local, temporary)</p></li>
                    110: </ul></div>
                    111: </div>
                    112: <div class="sect1" lang="en">
                    113: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
                    114: <a name="bulk.old"></a>7.3. Running an old-style bulk build</h2></div></div></div>
                    115: <div class="warning" style="margin-left: 0.5in; margin-right: 0.5in;">
                    116: <h3 class="title">Warning</h3>
                    117: <p>The rest of this section is rather old. Don't rely on it
                    118: too much.</p>
                    119: </div>
                    120: <div class="sect2" lang="en">
                    121: <div class="titlepage"><div><div><h3 class="title">
                    122: <a name="binary.configuration"></a>7.3.1. Configuration</h3></div></div></div>
                    123: <div class="sect3" lang="en">
                    124: <div class="titlepage"><div><div><h4 class="title">
                    125: <a name="binary.bulk.build.conf"></a>7.3.1.1. <code class="filename">build.conf</code>
                    126: </h4></div></div></div>
                    127: <p>The <code class="filename">build.conf</code> file is the main
                    128:        configuration file for bulk builds. You can configure how your
                    129:        copy of pkgsrc is kept up to date, how the distfiles are
                    130:        downloaded, how the packages are built and how the report is
                    131:        generated. You can find an annotated example file in
                    132:        <code class="filename">pkgsrc/mk/bulk/build.conf-example</code>. To use
                    133:        it, copy <code class="filename">build.conf-example</code> to
                    134:        <code class="filename">build.conf</code> and edit it, following the
                    135:        comments in that file.</p>
                    136: </div>
                    137: <div class="sect3" lang="en">
                    138: <div class="titlepage"><div><div><h4 class="title">
1.2     ! hubertf   139: <a name="binary.mk.conf"></a>7.3.1.2. <a class="link" href="configuring.html#mk.conf"><code class="filename">mk.conf</code></a>
1.1       rillig    140: </h4></div></div></div>
1.2     ! hubertf   141: <p>You may want to set variables in <a class="link" href="configuring.html#mk.conf"><code class="filename">mk.conf</code></a>.
1.1       rillig    142:        Look at <code class="filename">pkgsrc/mk/defaults/mk.conf</code> for
                    143:        details of the default settings. You will want to ensure that
                    144:        <code class="varname">ACCEPTABLE_LICENSES</code> meet your local policy.
                    145:        As used in this example, <code class="varname">_ACCEPTABLE=yes</code>
                    146:        completely bypasses the license check.</p>
                    147: <pre class="programlisting">
                    148: PACKAGES?=      ${_PKGSRCDIR}/packages/${MACHINE_ARCH}
                    149: WRKOBJDIR?=     /usr/tmp/pkgsrc   # build here instead of in pkgsrc
                    150: BSDSRCDIR=      /usr/src
                    151: BSDXSRCDIR=     /usr/xsrc         # for x11/xservers
                    152: OBJHOSTNAME?=   yes               # use work.`hostname`
                    153: FAILOVER_FETCH= yes               # insist on the correct checksum
                    154: PKG_DEVELOPER?= yes
                    155: _ACCEPTABLE=    yes
                    156: </pre>
                    157: <p>Some options that are especially useful for bulk builds
                    158:        can be found at the top lines of the file
                    159:        <code class="filename">mk/bulk/bsd.bulk-pkg.mk</code>. The most useful
                    160:        options of these are briefly described here.</p>
                    161: <div class="itemizedlist"><ul type="disc">
                    162: <li><p>If you are on a slow machine, you may want to
                    163:          set <code class="varname">USE_BULK_BROKEN_CHECK</code> to
                    164:          &#8220;<span class="quote">no</span>&#8221;.</p></li>
                    165: <li><p>If you are doing bulk builds from a read-only
                    166:          copy of pkgsrc, you have to set <code class="varname">BULKFILESDIR</code>
                    167:          to the directory where all log files are created. Otherwise the
                    168:          log files are created in the pkgsrc directory.</p></li>
                    169: <li><p>Another important variable is
                    170:          <code class="varname">BULK_PREREQ</code>, which is a list of packages that
                    171:          should be always available while building other
                    172:          packages.</p></li>
                    173: </ul></div>
                    174: <p>Some other options are scattered in the pkgsrc
                    175:        infrastructure:</p>
                    176: <div class="itemizedlist"><ul type="disc">
                    177: <li><p><code class="varname">ALLOW_VULNERABLE_PACKAGES</code>
                    178:          should be set to <code class="literal">yes</code>. The purpose of the bulk
                    179:          builds is creating binary packages, no matter if they are
                    180:          vulnerable or not. When uploading the packages to a public
                    181:          server, the vulnerable packages will be put into a directory of
                    182:          their own. Leaving this variable unset would prevent the bulk
                    183:          build system from even trying to build them, so possible
                    184:          building errors would not show up.</p></li>
                    185: <li><p><code class="varname">CHECK_FILES</code>
                    186:          (<code class="filename">pkgsrc/mk/check/check-files.mk</code>) can be set to
                    187:          &#8220;<span class="quote">yes</span>&#8221; to check that the installed set of files
                    188:          matches the <code class="filename">PLIST</code>.</p></li>
                    189: <li><p><code class="varname">CHECK_INTERPRETER</code>
                    190:          (<code class="filename">pkgsrc/mk/check/check-interpreter.mk</code>) can be set to
                    191:          &#8220;<span class="quote">yes</span>&#8221; to check that the installed
                    192:          &#8220;<span class="quote">#!</span>&#8221;-scripts will find their
                    193:          interpreter.</p></li>
                    194: <li><p><code class="varname">PKGSRC_RUN_TEST</code> can be
                    195:          set to &#8220;<span class="quote"><code class="literal">yes</code></span>&#8221; to run each
                    196:          package's self-test before installing it. Note that some
                    197:          packages make heavy use of &#8220;<span class="quote">good</span>&#8221; random
                    198:          numbers, so you need to assure that the machine on which you
                    199:          are doing the bulk builds is not completely idle. Otherwise
                    200:          some test programs will seem to hang, while they are just
                    201:          waiting for new random data to be
                    202:          available.</p></li>
                    203: </ul></div>
                    204: </div>
                    205: <div class="sect3" lang="en">
                    206: <div class="titlepage"><div><div><h4 class="title">
                    207: <a name="pre-build.local"></a>7.3.1.3. <code class="filename">pre-build.local</code>
                    208: </h4></div></div></div>
                    209: <p>It is possible to configure the bulk build to perform
                    210:        certain site-specific tasks at the end of the pre-build
                    211:        stage.  If the file
                    212:        <code class="filename">pre-build.local</code> exists in
                    213:        <code class="filename">/usr/pkgsrc/mk/bulk</code>, it will be executed
1.2     ! hubertf   214:        (as a <a class="citerefentry" href="http://netbsd.gw.com/cgi-bin/man-cgi?sh+1+NetBSD-current"><span class="citerefentry"><span class="refentrytitle">sh</span>(1)</span></a> script) at the end of the usual pre-build
1.1       rillig    215:        stage.  An example use of
                    216:        <code class="filename">pre-build.local</code> is to have the line:</p>
                    217: <pre class="screen">echo "I do not have enough disk space to build this pig." \
                    218:        &gt; misc/openoffice/$BROKENF</pre>
                    219: <p>to prevent the system from trying to build a particular package
                    220:        which requires nearly 3 GB of disk space.</p>
                    221: </div>
                    222: </div>
                    223: <div class="sect2" lang="en">
                    224: <div class="titlepage"><div><div><h3 class="title">
                    225: <a name="other-environmental-considerations"></a>7.3.2. Other environmental considerations</h3></div></div></div>
                    226: <p>As <code class="filename">/usr/pkg</code> will be completely
                    227:       deleted at the start of bulk builds, make sure your login
                    228:       shell is placed somewhere else. Either drop it into
                    229:       <code class="filename">/usr/local/bin</code> (and adjust your login
                    230:       shell in the passwd file), or (re-)install it via
1.2     ! hubertf   231:       <a class="citerefentry" href="http://netbsd.gw.com/cgi-bin/man-cgi?pkg_add+1+NetBSD-current"><span class="citerefentry"><span class="refentrytitle">pkg_add</span>(1)</span></a> from <code class="filename">/etc/rc.local</code>, so
1.1       rillig    232:       you can login after a reboot (remember that your current
                    233:       process won't die if the package is removed, you just can't
                    234:       start any new instances of the shell any more).  Also, if you
                    235:       use NetBSD earlier than 1.5, or you still want to use the pkgsrc
                    236:       version of ssh for some reason, be sure to install ssh before
                    237:       starting it from <code class="filename">rc.local</code>:</p>
                    238: <pre class="programlisting">
                    239: (cd /usr/pkgsrc/security/ssh &amp;&amp; make bulk-install)
                    240: if [ -f /usr/pkg/etc/rc.d/sshd ]; then
                    241:   /usr/pkg/etc/rc.d/sshd
                    242: fi
                    243: </pre>
                    244: <p>Not doing so will result in you being not able to log in
                    245:       via ssh after the bulk build is finished or if the machine
                    246:       gets rebooted or crashes. You have been warned! :)</p>
                    247: </div>
                    248: <div class="sect2" lang="en">
                    249: <div class="titlepage"><div><div><h3 class="title">
                    250: <a name="operation"></a>7.3.3. Operation</h3></div></div></div>
                    251: <p>Make sure you don't need any of the packages still
                    252:       installed.</p>
                    253: <div class="warning" style="margin-left: 0.5in; margin-right: 0.5in;">
                    254: <h3 class="title">Warning</h3>
                    255: <p>During the bulk build, <span class="emphasis"><em>all packages, their
                    256:        configuration files and some more files from
                    257:        <code class="filename">/var</code>, <code class="filename">/home</code> and
                    258:        possibly other locations will be removed! So don't run a bulk
                    259:        build with privileges that might harm your
                    260:        system.</em></span></p>
                    261: </div>
                    262: <p>Be sure to remove all other things that might
                    263:       interfere with builds, like some libs installed in
                    264:       <code class="filename">/usr/local</code>, etc. then become root and type:</p>
                    265: <pre class="screen">
                    266: <code class="prompt">#</code> <strong class="userinput"><code>cd /usr/pkgsrc</code></strong>
                    267: <code class="prompt">#</code> <strong class="userinput"><code>sh mk/bulk/build</code></strong>
                    268:       </pre>
                    269: <p>If for some reason your last build didn't complete (power
                    270:       failure, system panic, ...), you can continue it by
                    271:       running:</p>
                    272: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>sh mk/bulk/build restart</code></strong></pre>
                    273: <p>At the end of the bulk build, you will get a summary via mail,
                    274:       and find build logs in the directory specified by
                    275:       <code class="varname">FTP</code> in the <code class="filename">build.conf</code>
                    276:       file.</p>
                    277: </div>
                    278: <div class="sect2" lang="en">
                    279: <div class="titlepage"><div><div><h3 class="title">
                    280: <a name="what-it-does"></a>7.3.4. What it does</h3></div></div></div>
                    281: <p>The bulk builds consist of three steps:</p>
                    282: <div class="variablelist"><dl>
                    283: <dt><span class="term">1. pre-build</span></dt>
                    284: <dd><p>The script updates your pkgsrc tree via (anon)cvs, then
                    285:            cleans out any broken distfiles, and removes all
                    286:            packages installed.</p></dd>
                    287: <dt><span class="term">2. the bulk build</span></dt>
                    288: <dd><p>This is basically &#8220;<span class="quote">make bulk-package</span>&#8221; with
                    289:            an optimised order in which packages will be
                    290:            built. Packages that don't require other packages will
                    291:            be built first, and packages with many dependencies will
                    292:            be built later.</p></dd>
                    293: <dt><span class="term">3. post-build</span></dt>
                    294: <dd><p>Generates a report that's placed in the directory
                    295:            specified in the <code class="filename">build.conf</code> file
                    296:            named <code class="filename">broken.html</code>, a short version
                    297:            of that report will also be mailed to the build's
                    298:            admin.</p></dd>
                    299: </dl></div>
                    300: <p>During the build, a list of broken packages will be compiled
                    301:       in <code class="filename">/usr/pkgsrc/.broken</code> (or
                    302:       <code class="filename">.../.broken.${MACHINE}</code> if
                    303:       <code class="varname">OBJMACHINE</code> is set), individual build logs
                    304:       of broken builds can be found in the package's
                    305:       directory. These files are used by the bulk-targets to mark
                    306:       broken builds to not waste time trying to rebuild them, and
                    307:       they can be used to debug these broken package builds
                    308:       later.</p>
                    309: </div>
                    310: <div class="sect2" lang="en">
                    311: <div class="titlepage"><div><div><h3 class="title">
                    312: <a name="disk-space-requirements"></a>7.3.5. Disk space requirements</h3></div></div></div>
                    313: <p>Currently, roughly the following requirements are valid for
                    314:       NetBSD 2.0/i386:</p>
                    315: <div class="itemizedlist"><ul type="disc">
                    316: <li><p>10 GB - distfiles (NFS ok)</p></li>
                    317: <li><p>8 GB - full set of all binaries (NFS ok)</p></li>
                    318: <li><p>5 GB - temp space for compiling (local disk recommended)</p></li>
                    319: </ul></div>
                    320: <p>Note that all pkgs will be de-installed as soon as they are
                    321:       turned into a binary package, and that sources are removed,
                    322:       so there is no excessively huge demand to disk
                    323:       space. Afterwards, if the package is needed again, it will
1.2     ! hubertf   324:       be installed via <a class="citerefentry" href="http://netbsd.gw.com/cgi-bin/man-cgi?pkg_add+1+NetBSD-current"><span class="citerefentry"><span class="refentrytitle">pkg_add</span>(1)</span></a> instead of building again, so
1.1       rillig    325:       there are no cycles wasted by recompiling.</p>
                    326: </div>
                    327: <div class="sect2" lang="en">
                    328: <div class="titlepage"><div><div><h3 class="title">
                    329: <a name="setting-up-a-sandbox"></a>7.3.6. Setting up a sandbox for chrooted builds</h3></div></div></div>
                    330: <p>If you don't want all the packages nuked from a machine
                    331:       (rendering it useless for anything but pkg compiling), there
                    332:       is the possibility of doing the package bulk build inside a
                    333:       chroot environment.</p>
                    334: <p>The first step is to set up a chroot sandbox,
                    335:       e.g. <code class="filename">/usr/sandbox</code>.  This can be done by
                    336:       using null mounts, or manually.</p>
                    337: <p>There is a shell script called
                    338:       <code class="filename">pkgsrc/mk/bulk/mksandbox</code> which will set
                    339:       up the sandbox environment using null mounts. It will also
                    340:       create a script called <code class="filename">sandbox</code> in the
                    341:       root of the sandbox environment, which will allow the null
1.2     ! hubertf   342:       mounts to be activated using the <span class="command"><strong>sandbox
1.1       rillig    343:       mount</strong></span> command and deactivated using the
1.2     ! hubertf   344:       <span class="command"><strong>sandbox umount</strong></span> command.</p>
1.1       rillig    345: <p>To set up a sandbox environment by hand, after extracting all
1.2     ! hubertf   346:       the sets from a NetBSD installation or doing a <span class="command"><strong>make
1.1       rillig    347:       distribution DESTDIR=/usr/sandbox</strong></span> in
                    348:       <code class="filename">/usr/src/etc</code>, be sure the following items
                    349:       are present and properly configured:</p>
                    350: <div class="procedure"><ol type="1">
                    351: <li>
                    352: <p>Kernel</p>
                    353: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>cp /netbsd /usr/sandbox</code></strong></pre>
                    354: </li>
                    355: <li>
                    356: <p><code class="filename">/dev/*</code></p>
                    357: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>cd /usr/sandbox/dev ; sh MAKEDEV all</code></strong></pre>
                    358: </li>
                    359: <li>
                    360: <p><code class="filename">/etc/resolv.conf</code> (for <a href="ftp://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc/security/smtpd/README.html" target="_top"><code class="filename">security/smtpd</code></a> and mail):</p>
                    361: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>cp /etc/resolv.conf /usr/sandbox/etc</code></strong></pre>
                    362: </li>
                    363: <li>
                    364: <p>Working(!) mail config (hostname, sendmail.cf):</p>
                    365: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>cp /etc/mail/sendmail.cf /usr/sandbox/etc/mail</code></strong></pre>
                    366: </li>
                    367: <li>
                    368: <p><code class="filename">/etc/localtime</code> (for <a href="ftp://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc/security/smtpd/README.html" target="_top"><code class="filename">security/smtpd</code></a>):</p>
                    369: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>ln -sf /usr/share/zoneinfo/UTC /usr/sandbox/etc/localtime</code></strong></pre>
                    370: </li>
                    371: <li>
                    372: <p><code class="filename">/usr/src</code> (system sources,
1.2     ! hubertf   373:          e. g. for <a href="ftp://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc/sysutils/aperture/README.html" target="_top"><code class="filename">sysutils/aperture</code></a>):</p>
1.1       rillig    374: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>ln -s ../disk1/cvs .</code></strong>
                    375:          <code class="prompt">#</code> <strong class="userinput"><code>ln -s cvs/src-2.0 src</code></strong></pre>
                    376: </li>
                    377: <li>
                    378: <p>Create <code class="filename">/var/db/pkg</code> (not part of default install):</p>
                    379: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>mkdir /usr/sandbox/var/db/pkg</code></strong></pre>
                    380: </li>
                    381: <li>
                    382: <p>Create <code class="filename">/usr/pkg</code> (not part of default install):</p>
                    383: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>mkdir /usr/sandbox/usr/pkg</code></strong></pre>
                    384: </li>
                    385: <li>
                    386: <p>Checkout pkgsrc via cvs into
                    387:          <code class="filename">/usr/sandbox/usr/pkgsrc</code>:</p>
                    388: <pre class="screen">
                    389: <code class="prompt">#</code> <strong class="userinput"><code>cd /usr/sandbox/usr</code></strong>
                    390: <code class="prompt">#</code> <strong class="userinput"><code>cvs -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -d -P pkgsrc</code></strong>
                    391:          </pre>
                    392: <p>Do not mount/link this to the copy of your pkgsrc tree
                    393:          you do development in, as this will likely cause problems!</p>
                    394: </li>
                    395: <li><p>Make
                    396:          <code class="filename">/usr/sandbox/usr/pkgsrc/packages</code> and
                    397:          <code class="filename">.../distfiles</code> point somewhere
                    398:          appropriate. NFS- and/or nullfs-mounts may come in handy!</p></li>
1.2     ! hubertf   399: <li><p>Edit <a class="link" href="configuring.html#mk.conf"><code class="filename">mk.conf</code></a>, see <a class="xref" href="bulk.html#binary.mk.conf" title="7.3.1.2. mk.conf">Section 7.3.1.2, &#8220;<code class="filename">mk.conf</code>&#8221;</a>.</p></li>
1.1       rillig    400: <li><p>Adjust <code class="filename">mk/bulk/build.conf</code> to suit your needs.</p></li>
                    401: </ol></div>
                    402: <p>When the chroot sandbox is set up, you can start
                    403:       the build with the following steps:</p>
                    404: <pre class="screen">
                    405: <code class="prompt">#</code> <strong class="userinput"><code>cd /usr/sandbox/usr/pkgsrc</code></strong>
                    406: <code class="prompt">#</code> <strong class="userinput"><code>sh mk/bulk/do-sandbox-build</code></strong>
                    407:       </pre>
                    408: <p>This will just jump inside the sandbox and start building.  At
                    409:        the end of the build, mail will be sent with the results of
                    410:        the build. Created binary pkgs will be in
                    411:        <code class="filename">/usr/sandbox/usr/pkgsrc/packages</code>
                    412:       (wherever that points/mounts to/from).</p>
                    413: </div>
                    414: <div class="sect2" lang="en">
                    415: <div class="titlepage"><div><div><h3 class="title">
                    416: <a name="building-a-partial-set"></a>7.3.7. Building a partial set of packages</h3></div></div></div>
                    417: <p>In addition to building a complete set of all packages in
                    418:       pkgsrc, the <code class="filename">pkgsrc/mk/bulk/build</code> script
                    419:       may be used to build a subset of the packages contained in
                    420:       pkgsrc.  By setting <code class="varname">SPECIFIC_PKGS</code>
1.2     ! hubertf   421:       in <a class="link" href="configuring.html#mk.conf"><code class="filename">mk.conf</code></a>, the variables</p>
1.1       rillig    422: <div class="itemizedlist"><ul type="disc">
                    423: <li><p>SITE_SPECIFIC_PKGS</p></li>
                    424: <li><p>HOST_SPECIFIC_PKGS</p></li>
                    425: <li><p>GROUP_SPECIFIC_PKGS</p></li>
                    426: <li><p>USER_SPECIFIC_PKGS</p></li>
                    427: </ul></div>
                    428: <p>will define the set of packages which should be built.
                    429:       The bulk build code will also include any packages which are
                    430:       needed as dependencies for the explicitly listed packages.</p>
                    431: <p>One use of this is to do a bulk build with
                    432:       <code class="varname">SPECIFIC_PKGS</code> in a chroot sandbox
                    433:       periodically to have a complete set of the binary packages
                    434:       needed for your site available without the overhead of
                    435:       building extra packages that are not needed.</p>
                    436: </div>
                    437: <div class="sect2" lang="en">
                    438: <div class="titlepage"><div><div><h3 class="title">
                    439: <a name="bulk-upload"></a>7.3.8. Uploading results of a bulk build</h3></div></div></div>
                    440: <p>This section describes how pkgsrc developers can upload binary
                    441:       pkgs built by bulk builds to ftp.NetBSD.org.</p>
                    442: <p>If you would like to automatically create checksum files for the
                    443:       binary packages you intend to upload, remember to set
                    444:       <code class="varname">MKSUMS=yes</code> in your
                    445:       <code class="filename">mk/bulk/build.conf</code>.</p>
                    446: <p>If you would like to PGP sign the checksum files (highly
                    447:       recommended!), remember to set
                    448:       <code class="varname">SIGN_AS=username@NetBSD.org</code> in your
                    449:       <code class="filename">mk/bulk/build.conf</code>.  This will prompt you for
                    450:       your GPG password to sign the files before uploading everything.</p>
                    451: <p>Then, make sure that you have <code class="varname">RSYNC_DST</code>
                    452:       set properly in your <code class="filename">mk/bulk/build.conf</code>
                    453:       file, i.e. adjust it to something like one of the following:</p>
                    454: <pre class="screen">RSYNC_DST=ftp.NetBSD.org:/pub/NetBSD/packages/packages-200xQy/NetBSD-a.b.c/arch/upload </pre>
                    455: <p>Please use appropriate values for "packages-200xQy",
                    456:       "NetBSD-a.b.c" and "arch" here. If your login on ftp.NetBSD.org
                    457:       is different from your local login, write your login directly
                    458:       into the variable, e.g. my local account is "feyrer", but for my
                    459:       login "hubertf", I use:</p>
                    460: <pre class="screen">RSYNC_DST=hubertf@ftp.NetBSD.org:/pub/NetBSD/packages/packages-200xQy/NetBSD-a.b.c/arch/upload</pre>
                    461: <p>A separate <code class="filename">upload</code> directory is used
                    462:       here to allow "closing" the directory during upload. To do
                    463:       so, run the following command on ftp.NetBSD.org next:</p>
                    464: <pre class="screen">nbftp% <strong class="userinput"><code>mkdir -p -m 750 /pub/NetBSD/packages/packages-200xQy/NetBSD-a.b.c/arch/upload</code></strong></pre>
                    465: <p>Please note that <code class="filename">/pub/NetBSD/packages</code> is
                    466:       only appropriate for packages for the NetBSD operating
                    467:       system. Binary packages for other operating systems should go
                    468:       into <code class="filename">/pub/pkgsrc</code>.</p>
                    469: <p>Before uploading the binary pkgs, ssh authentication needs
                    470:       to be set up. This example shows how to set up temporary keys
                    471:       for the root account <span class="emphasis"><em>inside the sandbox</em></span>
                    472:       (assuming that no keys should be present there usually):</p>
                    473: <pre class="screen">
                    474: <code class="prompt">#</code> <strong class="userinput"><code>chroot /usr/sandbox</code></strong>
                    475: chroot-<code class="prompt">#</code> <strong class="userinput"><code>rm $HOME/.ssh/id-dsa*</code></strong>
                    476: chroot-<code class="prompt">#</code> <strong class="userinput"><code>ssh-keygen -t dsa</code></strong>
                    477: chroot-<code class="prompt">#</code> <strong class="userinput"><code>cat $HOME/.ssh/id-dsa.pub</code></strong>
                    478:       </pre>
                    479: <p>Now take the output of <code class="filename">id-dsa.pub</code> and
                    480:       append it to your <code class="filename">~/.ssh/authorized_keys</code>
                    481:       file on ftp.NetBSD.org. You can remove the key after the
                    482:       upload is done!</p>
                    483: <p>Next, test if your ssh connection really works:</p>
                    484: <pre class="screen">chroot-<code class="prompt">#</code> <strong class="userinput"><code>ssh ftp.NetBSD.org date</code></strong> </pre>
                    485: <p>Use "-l yourNetBSDlogin" here as appropriate!</p>
                    486: <p>Now after all this works, you can exit the sandbox and start
                    487:       the upload:</p>
                    488: <pre class="screen">
                    489: chroot-<code class="prompt">#</code> <strong class="userinput"><code>exit</code></strong>
                    490: <code class="prompt">#</code> <strong class="userinput"><code>cd /usr/sandbox/usr/pkgsrc</code></strong>
                    491: <code class="prompt">#</code> <strong class="userinput"><code>sh mk/bulk/do-sandbox-upload</code></strong>
                    492:       </pre>
1.2     ! hubertf   493: <p>The upload process may take quite some time. Use <a class="citerefentry" href="http://netbsd.gw.com/cgi-bin/man-cgi?ls+1+NetBSD-current"><span class="citerefentry"><span class="refentrytitle">ls</span>(1)</span></a> or
        !           494:       <a class="citerefentry" href="http://netbsd.gw.com/cgi-bin/man-cgi?du+1+NetBSD-current"><span class="citerefentry"><span class="refentrytitle">du</span>(1)</span></a> on the FTP server to monitor progress of the
1.1       rillig    495:       upload. The upload script will take care of not uploading
                    496:       restricted packages and putting vulnerable packages into the
                    497:       <code class="filename">vulnerable</code> subdirectory.</p>
                    498: <p>After the upload has ended, first thing is to revoke ssh access:</p>
                    499: <pre class="screen">nbftp% <strong class="userinput"><code>vi ~/.ssh/authorized_keys</code></strong>
                    500:       Gdd:x! </pre>
                    501: <p>Use whatever is needed to remove the key you've entered
                    502:       before! Last, move the uploaded packages out of the
                    503:       <code class="filename">upload</code> directory to have them accessible
                    504:       to everyone:</p>
                    505: <pre class="screen">
                    506: nbftp% <strong class="userinput"><code>cd /pub/NetBSD/packages/packages-200xQy/NetBSD-a.b.c/arch</code></strong>
                    507: nbftp% <strong class="userinput"><code>mv upload/* .</code></strong>
                    508: nbftp% <strong class="userinput"><code>rmdir upload</code></strong>
                    509: nbftp% <strong class="userinput"><code>chmod 755 .</code></strong>
                    510:       </pre>
                    511: </div>
                    512: </div>
                    513: <div class="sect1" lang="en">
                    514: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
                    515: <a name="bulk.pbulk"></a>7.4. Running a pbulk-style bulk build</h2></div></div></div>
                    516: <div class="sect2" lang="en">
                    517: <div class="titlepage"><div><div><h3 class="title">
                    518: <a name="bulk.pbulk.conf"></a>7.4.1. Configuration</h3></div></div></div>
1.2     ! hubertf   519: <p>TODO; see <a class="ulink" href="http://wiki.netbsd.se/index.php/pbulk-HOWTO" target="_top">the wiki</a> for
1.1       rillig    520: more information.</p>
                    521: </div>
                    522: </div>
                    523: <div class="sect1" lang="en">
                    524: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
                    525: <a name="creating-cdroms"></a>7.5. Creating a multiple CD-ROM packages collection</h2></div></div></div>
                    526: <p>After your pkgsrc bulk-build has completed, you may wish to
                    527:     create a CD-ROM set of the resulting binary packages to assist
                    528:     in installing packages on other machines.  The
                    529:     <a href="ftp://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc/pkgtools/cdpack/README.html" target="_top"><code class="filename">pkgtools/cdpack</code></a> package provides
                    530:     a simple tool for creating the ISO 9660 images.
1.2     ! hubertf   531:     <span class="command"><strong>cdpack</strong></span> arranges the packages on the CD-ROMs in a
1.1       rillig    532:     way that keeps all the dependencies for a given package on the same
                    533:     CD as that package.</p>
                    534: <div class="sect2" lang="en">
                    535: <div class="titlepage"><div><div><h3 class="title">
                    536: <a name="cdpack-example"></a>7.5.1. Example of cdpack</h3></div></div></div>
                    537: <p>Complete documentation for cdpack is found in the cdpack(1)
                    538:       man page. The following short example assumes that the binary
                    539:       packages are left in
                    540:       <code class="filename">/usr/pkgsrc/packages/All</code> and that
                    541:       sufficient disk space exists in <code class="filename">/u2</code> to
                    542:       hold the ISO 9660 images.</p>
                    543: <pre class="screen">
                    544: <code class="prompt">#</code> <strong class="userinput"><code>mkdir /u2/images</code></strong>
                    545: <code class="prompt">#</code> <strong class="userinput"><code>pkg_add /usr/pkgsrc/packages/All/cdpack</code></strong>
                    546: <code class="prompt">#</code> <strong class="userinput"><code>cdpack /usr/pkgsrc/packages/All /u2/images</code></strong>
                    547:       </pre>
                    548: <p>If you wish to include a common set of files
                    549:       (<code class="filename">COPYRIGHT</code>, <code class="filename">README</code>,
                    550:       etc.) on each CD in the collection, then you need to create a
                    551:       directory which contains these files. e.g.</p>
                    552: <pre class="screen">
                    553: <code class="prompt">#</code> <strong class="userinput"><code>mkdir /tmp/common</code></strong>
                    554: <code class="prompt">#</code> <strong class="userinput"><code>echo "This is a README" &gt; /tmp/common/README</code></strong>
                    555: <code class="prompt">#</code> <strong class="userinput"><code>echo "Another file" &gt; /tmp/common/COPYING</code></strong>
                    556: <code class="prompt">#</code> <strong class="userinput"><code>mkdir /tmp/common/bin</code></strong>
                    557: <code class="prompt">#</code> <strong class="userinput"><code>echo "#!/bin/sh" &gt; /tmp/common/bin/myscript</code></strong>
                    558: <code class="prompt">#</code> <strong class="userinput"><code>echo "echo Hello world" &gt;&gt; /tmp/common/bin/myscript</code></strong>
                    559: <code class="prompt">#</code> <strong class="userinput"><code>chmod 755 /tmp/common/bin/myscript</code></strong>
                    560:       </pre>
                    561: <p>Now create the images:</p>
                    562: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images</code></strong></pre>
                    563: <p>Each image will contain <code class="filename">README</code>,
                    564:       <code class="filename">COPYING</code>, and <code class="filename">bin/myscript</code>
                    565:       in their root directories.</p>
                    566: </div>
                    567: </div>
                    568: </div>
                    569: <div class="navfooter">
                    570: <hr>
                    571: <table width="100%" summary="Navigation footer">
                    572: <tr>
                    573: <td width="40%" align="left">
                    574: <a accesskey="p" href="binary.html">Prev</a> </td>
                    575: <td width="20%" align="center"><a accesskey="u" href="users-guide.html">Up</a></td>
                    576: <td width="40%" align="right"> <a accesskey="n" href="files.html">Next</a>
                    577: </td>
                    578: </tr>
                    579: <tr>
                    580: <td width="40%" align="left" valign="top">Chapter 6. Creating binary packages </td>
                    581: <td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td>
                    582: <td width="40%" align="right" valign="top"> Chapter 8. Directory layout of the installed files</td>
                    583: </tr>
                    584: </table>
                    585: </div>
                    586: </body>
                    587: </html>

CVSweb <webmaster@jp.NetBSD.org>