Annotation of pkgsrc/doc/guide/files/binary.xml, revision 1.22

1.22    ! rillig      1: <!-- $NetBSD: binary.xml,v 1.21 2006/02/12 14:44:59 rillig Exp $ -->
1.1       grant       2: 
                      3: <chapter id="binary">
                      4:   <title>Creating binary packages</title>
                      5: 
1.13      wiz         6:   <sect1 id="building-a-single-binary-package">
1.1       grant       7:     <title>Building a single binary package</title>
                      8: 
1.6       hubertf     9:     <para>
                     10:       Once you have built and installed a package, you can create a
                     11:       <emphasis>binary package</emphasis> which can be installed on
1.15      rillig     12:       another system with &man.pkg.add.1;. This saves having to build
1.6       hubertf    13:       the same package on a group of hosts and wasting CPU time. It
                     14:       also provides a simple means for others to install your package,
                     15:       should you distribute it.
                     16:     </para>
                     17: 
                     18:     <para>
                     19:       To create a binary package, change into the appropriate
                     20:       directory in pkgsrc, and run <command>make
                     21:       package</command>:
                     22:     </para>
1.1       grant      23: 
                     24:     <screen>&rprompt; <userinput>cd misc/figlet</userinput>
                     25: &rprompt; <userinput>make package</userinput></screen>
                     26: 
1.6       hubertf    27:     <para>
                     28:       This will build and install your package (if not already done),
                     29:       and then build a binary package from what was installed. You can
                     30:       then use the <command>pkg_*</command> tools to manipulate
                     31:       it. Binary packages are created by default in
                     32:       <filename>/usr/pkgsrc/packages</filename>, in the form of a
                     33:       gzipped tar file. See <xref linkend="logs.package"/> for a
1.21      rillig     34:       continuation of the above <filename
1.14      rpaulo     35:       role="pkg">misc/figlet</filename> example.</para>
1.6       hubertf    36: 
                     37:     <para>
                     38:       See <xref linkend="submit"/> for information on how to submit
                     39:       such a binary package.</para>
1.1       grant      40:   </sect1>
                     41: 
1.13      wiz        42:   <sect1 id="settings-for-creationg-of-binary-packages">
1.1       grant      43:     <title>Settings for creation of binary packages</title>
                     44: 
                     45:     <para>See <xref linkend="build.helpful-targets"/>.</para>
                     46:   </sect1>
                     47: 
                     48:   <sect1 id="bulkbuild">
                     49:     <title>Doing a bulk build of all packages</title>
                     50: 
1.9       wiz        51:     <para>If you want to get a full set of precompiled binary
                     52:       packages, this section describes how to get them. Beware that
                     53:       the bulk build will remove all currently installed packages from
1.17      rillig     54:       your system!</para>
                     55: 
                     56:       <para>Having an FTP server configured either on the
1.9       wiz        57:       machine doing the bulk builds or on a nearby NFS server can help
1.17      rillig     58:       to make the packages available to other machines that can then
                     59:       save time by installing only the binary packages. See &man.ftpd.8; for
1.9       wiz        60:       more information. If you use a remote NFS server's storage, be
                     61:       sure to not actually compile on NFS storage, as this slows
                     62:       things down a lot.</para>
1.1       grant      63: 
                     64:     <sect2 id="binary.configuration">
                     65:       <title>Configuration</title>
                     66: 
1.17      rillig     67:       <sect3 id="binary.bulk.build.conf">
                     68:        <title><filename>build.conf</filename></title>
                     69: 
                     70:         <para>The <filename>build.conf</filename> file is the main
                     71:         configuration file for bulk builds. You can configure how your
                     72:         copy of pkgsrc is kept up to date, how the distfiles are
                     73:         downloaded, how the packages are built and how the report is
                     74:         generated. You can find an annotated example file in
                     75:         <filename>pkgsrc/mk/bulk/build.conf-example</filename>. To use
                     76:         it, copy <filename>build.conf-example</filename> to
                     77:         <filename>build.conf</filename> and edit it, following the
                     78:         comments in that file.</para>
                     79:       </sect3>
                     80: 
1.1       grant      81:       <sect3 id="binary.mk.conf">
                     82:        <title>/etc/mk.conf</title>
                     83: 
1.9       wiz        84:        <para>You may want to set variables in
                     85:          <filename>/etc/mk.conf</filename>.
1.2       ben        86:          Look at <filename>pkgsrc/mk/defaults/mk.conf</filename> for
1.1       grant      87:          details of the default settings. You will want to ensure that
                     88:          <varname>ACCEPTABLE_LICENSES</varname> meet your local policy.
                     89:          As used in this example, <varname>_ACCEPTABLE=yes</varname>
                     90:          accepts <emphasis>all</emphasis> licenses.</para>
                     91: 
1.16      rillig     92: <programlisting>
                     93:     PACKAGES?=      ${_PKGSRCDIR}/packages/${MACHINE_ARCH}
                     94:     WRKOBJDIR?=     /usr/tmp/pkgsrc   # build here instead of in pkgsrc
                     95:     BSDSRCDIR=      /usr/src
                     96:     BSDXSRCDIR=     /usr/xsrc         # for x11/xservers
                     97:     OBJHOSTNAME?=   yes               # use work.`hostname`
                     98:     FAILOVER_FETCH= yes               # insist on the correct checksum
                     99:     PKG_DEVELOPER?= yes
                    100:     _ACCEPTABLE=    yes
                    101: </programlisting>
1.18      rillig    102: 
                    103:        <para>Some options that are especially useful for bulk builds
                    104:        can be found at the top lines of the file
                    105:        <filename>mk/bulk/bsd.bulk-pkg.mk</filename>. The most useful
                    106:        options of these are briefly described here.</para>
                    107: 
                    108:        <itemizedlist>
                    109: 
                    110:        <listitem><para>If you are on a slow machine, you may want to
                    111:        set <varname>USE_BULK_BROKEN_CHECK</varname> to
                    112:        <quote>no</quote>.</para></listitem>
                    113: 
                    114:        <listitem><para>If you are doing bulk builds from a read-only
                    115:        copy of pkgsrc, you have to set <varname>BULKFILESDIR</varname>
                    116:        to the directory where all log files are created. Otherwise the
                    117:        log files are created in the pkgsrc directory.</para></listitem>
                    118: 
                    119:        <listitem><para>Another important variable is
                    120:        <varname>BULK_PREREQ</varname>, which is a list of packages that
                    121:        should be always available while building other
                    122:        packages.</para></listitem>
                    123: 
                    124:        </itemizedlist>
1.19      rillig    125: 
                    126:        <para>Some other options are scattered in the pkgsrc
                    127:        infrastructure:</para>
                    128: 
                    129:        <itemizedlist>
                    130: 
1.22    ! rillig    131:        <listitem><para><varname>ALLOW_VULNERABLE_PACKAGES</varname>
        !           132:        should be set to <literal>yes</literal>. The purpose of the bulk
        !           133:        builds is creating binary packages, no matter if they are
        !           134:        vulnerable or not. When uploading the packages to a public
        !           135:        server, the vulnerable packages will be put into a directory of
        !           136:        their own. Leaving this variable unset would prevent the bulk
        !           137:        build system from even trying to build them, so possible
        !           138:        building errors would not show up.</para></listitem>
        !           139: 
1.20      rillig    140:        <listitem><para><varname>CHECK_FILES</varname>
                    141:        (<filename>pkgsrc/mk/bsd.pkg.check.mk</filename>) can be set to
1.19      rillig    142:        <quote>yes</quote> to check that the installed set of files
                    143:        matches the <filename>PLIST</filename>.</para></listitem>
                    144: 
1.20      rillig    145:        <listitem><para><varname>CHECK_INTERPRETER</varname>
                    146:        (<filename>pkgsrc/mk/bsd.pkg.check.mk</filename>) can be set to
                    147:        <quote>yes</quote> to check that the installed
1.19      rillig    148:        <quote>#!</quote>-scripts will find their
                    149:        interpreter.</para></listitem>
                    150: 
                    151:        </itemizedlist>
                    152: 
1.1       grant     153:       </sect3>
                    154: 
1.13      wiz       155:       <sect3 id="pre-build.local">
1.1       grant     156:        <title><filename>pre-build.local</filename></title>
                    157: 
1.9       wiz       158:        <para>It is possible to configure the bulk build to perform
1.15      rillig    159:          certain site-specific tasks at the end of the pre-build
1.21      rillig    160:          stage.  If the file
1.1       grant     161:          <filename>pre-build.local</filename> exists in
1.9       wiz       162:          <filename>/usr/pkgsrc/mk/bulk</filename>, it will be executed
1.11      wiz       163:          (as a &man.sh.1; script) at the end of the usual pre-build
1.9       wiz       164:          stage.  An example use of
                    165:          <filename>pre-build.local</filename> is to have the line:</para>
1.1       grant     166: 
                    167:        <screen>&rprompt; <userinput>echo "I do not have enough disk space to build this pig." \
1.9       wiz       168:     &gt; pkgsrc/misc/openoffice/$BROKENF</userinput></screen>
1.1       grant     169: 
                    170:        <para>to prevent the system from trying to build a particular package
                    171:          which requires nearly 3 GB of disk space.</para>
                    172:       </sect3>
                    173:     </sect2>
                    174: 
1.13      wiz       175:     <sect2 id="other-environmental-considerations">
1.1       grant     176:       <title>Other environmental considerations</title>
                    177: 
1.6       hubertf   178:       <para>As <filename>/usr/pkg</filename> will be completely
                    179:        deleted at the start of bulk builds, make sure your login
                    180:        shell is placed somewhere else. Either drop it into
                    181:        <filename>/usr/local/bin</filename> (and adjust your login
                    182:        shell in the passwd file), or (re-)install it via
                    183:        &man.pkg.add.1; from <filename>/etc/rc.local</filename>, so
                    184:        you can login after a reboot (remember that your current
                    185:        process won't die if the package is removed, you just can't
                    186:        start any new instances of the shell any more).  Also, if you
                    187:        use &os; earlier than 1.5, or you still want to use the pkgsrc
                    188:        version of ssh for some reason, be sure to install ssh before
1.1       grant     189:        starting it from <filename>rc.local</filename>:</para>
                    190: 
1.16      rillig    191: <programlisting>
                    192:     ( cd /usr/pkgsrc/security/ssh ; make bulk-install )
                    193:     if [ -f /usr/pkg/etc/rc.d/sshd ]; then
                    194:         /usr/pkg/etc/rc.d/sshd
                    195:     fi
                    196: </programlisting>
1.1       grant     197: 
1.6       hubertf   198:       <para>Not doing so will result in you being not able to log in
                    199:        via ssh after the bulk build is finished or if the machine
                    200:        gets rebooted or crashes. You have been warned! :)</para>
1.1       grant     201:     </sect2>
                    202: 
1.13      wiz       203:     <sect2 id="operation">
1.1       grant     204:       <title>Operation</title>
                    205: 
1.6       hubertf   206:       <para>Make sure you don't need any of the packages still
                    207:        installed.
1.21      rillig    208:       </para>
1.1       grant     209: 
                    210:       <warning>
1.21      rillig    211:        <para>During the bulk build, <emphasis>all packages will be
1.1       grant     212:            removed!</emphasis></para>
                    213:       </warning>
                    214: 
                    215:       <para>Be sure to remove all other things that might
                    216:        interfere with builds, like some libs installed in
                    217:        <filename>/usr/local</filename>, etc. then become root and type:</para>
                    218: 
                    219:       <screen>&rprompt; <userinput>cd /usr/pkgsrc</userinput>
                    220: &rprompt; <userinput>sh mk/bulk/build</userinput></screen>
                    221: 
1.6       hubertf   222:       <para>If for some reason your last build didn't complete (power
                    223:        failure, system panic, ...), you can continue it by
1.21      rillig    224:        running:</para>
1.1       grant     225: 
                    226:       <screen>&rprompt; <userinput>sh mk/bulk/build restart</userinput></screen>
                    227: 
1.6       hubertf   228:       <para>
                    229:         At the end of the bulk build, you will get a summary via mail,
                    230:        and find build logs in the directory specified by
                    231:        <varname>FTP</varname> in the <filename>build.conf</filename>
                    232:        file.
1.21      rillig    233:       </para>
1.1       grant     234:     </sect2>
                    235: 
1.13      wiz       236:     <sect2 id="what-it-does">
1.1       grant     237:       <title>What it does</title>
                    238: 
                    239:       <para>The bulk builds consist of three steps:</para>
                    240: 
                    241:       <variablelist>
                    242:        <varlistentry>
                    243:          <term>1. pre-build</term>
                    244: 
                    245:          <listitem>
1.6       hubertf   246:            <para>
                    247:              The script updates your pkgsrc tree via (anon)cvs, then
                    248:              cleans out any broken distfiles, and removes all
                    249:              packages installed.</para>
1.1       grant     250:          </listitem>
                    251:        </varlistentry>
                    252: 
                    253:        <varlistentry>
                    254:          <term>2. the bulk build</term>
                    255: 
                    256:          <listitem>
1.6       hubertf   257:            <para>
                    258:              This is basically <quote>make bulk-package</quote> with
                    259:              an optimised order in which packages will be
                    260:              built. Packages that don't require other packages will
                    261:              be built first, and packages with many dependencies will
                    262:              be built later.
                    263:            </para>
1.1       grant     264:          </listitem>
                    265:        </varlistentry>
                    266: 
                    267:        <varlistentry>
                    268:          <term>3. post-build</term>
                    269: 
                    270:          <listitem>
1.6       hubertf   271:            <para>
                    272:              Generates a report that's placed in the directory
                    273:              specified in the <filename>build.conf</filename> file
                    274:              named <filename>broken.html</filename>, a short version
                    275:              of that report will also be mailed to the build's
                    276:              admin.</para>
1.1       grant     277:          </listitem>
                    278:        </varlistentry>
                    279:       </variablelist>
                    280: 
1.6       hubertf   281:       <para>
                    282:         During the build, a list of broken packages will be compiled
                    283:         in <filename>/usr/pkgsrc/.broken</filename> (or
                    284:         <filename>.../.broken.${MACHINE}</filename> if
                    285:         <varname>OBJMACHINE</varname> is set), individual build logs
                    286:         of broken builds can be found in the package's
                    287:         directory. These files are used by the bulk-targets to mark
                    288:         broken builds  to not waste time trying to rebuild them, and
                    289:         they can be used to debug      these broken package builds
                    290:         later.
1.21      rillig    291:       </para>
1.1       grant     292:     </sect2>
                    293: 
1.13      wiz       294:     <sect2 id="disk-space-requirements">
1.1       grant     295:       <title>Disk space requirements</title>
                    296: 
                    297:       <para>Currently, roughly the following requirements are valid for
                    298:        NetBSD 2.0/i386:</para>
                    299: 
                    300:       <itemizedlist>
                    301:        <listitem>
1.6       hubertf   302:          <para>
                    303:            10 GB - distfiles (NFS ok)
                    304:          </para>
1.1       grant     305:        </listitem>
                    306: 
                    307:        <listitem>
1.6       hubertf   308:          <para>
                    309:            8 GB - full set of all binaries (NFS ok)
                    310:          </para>
1.1       grant     311:        </listitem>
                    312: 
                    313:        <listitem>
1.6       hubertf   314:          <para>
                    315:            5 GB - temp space for compiling (local disk recommended)
                    316:          </para>
1.1       grant     317:        </listitem>
                    318:       </itemizedlist>
                    319: 
1.6       hubertf   320:       <para>
                    321:         Note that all pkgs will be de-installed as soon as they are
                    322:         turned into a  binary package, and that sources are removed,
                    323:         so there is no excessively huge        demand to disk
                    324:         space. Afterwards, if the package is needed again, it will
                    325:         be installed via &man.pkg.add.1; instead of building again, so
                    326:         there are no cycles wasted by recompiling.
1.21      rillig    327:       </para>
1.1       grant     328:     </sect2>
                    329: 
1.13      wiz       330:     <sect2 id="setting-up-a-sandbox">
1.15      rillig    331:       <title>Setting up a sandbox for chrooted builds</title>
1.1       grant     332: 
1.6       hubertf   333:       <para>
                    334:         If you don't want all the packages nuked from a machine
                    335:         (rendering it useless for anything but pkg compiling), there
1.9       wiz       336:         is the possibility of doing the package bulk build inside a
1.6       hubertf   337:         chroot environment.
                    338:       </para>
1.21      rillig    339: 
1.6       hubertf   340:       <para>
                    341:         The first step is to set up a chroot sandbox,
                    342:        e.g. <filename>/usr/sandbox</filename>.  This can be done by
                    343:        using null mounts, or manually.
                    344:       </para>
                    345: 
                    346:       <para>
                    347:         There is a shell script called
                    348:         <filename>pkgsrc/mk/bulk/mksandbox</filename> which will set
                    349:         up the sandbox environment using null mounts. It will also
                    350:         create a script called <filename>sandbox</filename> in the
                    351:         root of the sandbox environment, which will allow the null
                    352:         mounts to be activated using the <command>sandbox
                    353:         mount</command> command and deactivated using the
                    354:         <command>sandbox umount</command> command.
1.21      rillig    355:       </para>
1.6       hubertf   356: 
                    357:       <para>
                    358:         To set up a sandbox environment by hand, after extracting all
                    359:         the sets from a &os; installation or doing a <command>make
                    360:         distribution DESTDIR=/usr/sandbox</command> in
                    361:         <filename>/usr/src/etc</filename>, be sure the following items
                    362:         are present and properly configured:
                    363:       </para>
1.1       grant     364: 
                    365:       <procedure>
                    366:        <step>
                    367:          <para>Kernel</para>
                    368: 
                    369:          <screen>&rprompt; <userinput>cp /netbsd /usr/sandbox</userinput></screen>
                    370:        </step>
                    371: 
                    372:        <step>
                    373:          <para><filename>/dev/*</filename></para>
                    374: 
                    375:          <screen>&rprompt; <userinput>cd /usr/sandbox/dev ; sh MAKEDEV all</userinput></screen>
                    376:        </step>
                    377: 
                    378:        <step>
1.21      rillig    379:          <para><filename>/etc/resolv.conf</filename> (for <filename
1.14      rpaulo    380:            role="pkg">security/smtpd</filename> and mail):</para>
1.1       grant     381: 
                    382:          <screen>&rprompt; <userinput>cp /etc/resolv.conf /usr/sandbox/etc</userinput></screen>
                    383:        </step>
                    384: 
                    385:        <step>
                    386:          <para>Working(!) mail config (hostname, sendmail.cf):</para>
                    387: 
                    388:          <screen>&rprompt; <userinput>cp /etc/mail/sendmail.cf /usr/sandbox/etc/mail</userinput></screen>
                    389:        </step>
                    390: 
                    391:        <step>
1.21      rillig    392:          <para><filename>/etc/localtime</filename> (for <filename
1.14      rpaulo    393:            role="pkg">security/smtpd</filename>):</para>
1.1       grant     394: 
                    395:          <screen>&rprompt; <userinput>ln -sf /usr/share/zoneinfo/UTC /usr/sandbox/etc/localtime</userinput></screen>
                    396:        </step>
                    397: 
                    398:        <step>
1.21      rillig    399:          <para><filename>/usr/src</filename> (system sources, for <filename
1.14      rpaulo    400:            role="pkg">sysutils/aperture</filename>,
                    401:            <filename role="pkg">net/ppp-mppe</filename>):</para>
1.1       grant     402: 
                    403:          <screen>&rprompt; <userinput>ln -s ../disk1/cvs .</userinput>
1.9       wiz       404: &rprompt; <userinput>ln -s cvs/src-2.0 src</userinput></screen>
1.1       grant     405:        </step>
                    406: 
                    407:        <step>
                    408:          <para>Create <filename>/var/db/pkg</filename> (not part of default install):</para>
                    409: 
                    410:          <screen>&rprompt; <userinput>mkdir /usr/sandbox/var/db/pkg</userinput></screen>
                    411:        </step>
                    412: 
                    413:        <step>
                    414:          <para>Create <filename>/usr/pkg</filename> (not part of default install):</para>
                    415: 
                    416:          <screen>&rprompt; <userinput>mkdir /usr/sandbox/usr/pkg</userinput></screen>
                    417:        </step>
                    418: 
                    419:        <step>
                    420:          <para>Checkout pkgsrc via cvs into
                    421:            <filename>/usr/sandbox/usr/pkgsrc</filename>:</para>
                    422: 
                    423:          <screen>&rprompt; <userinput>cd /usr/sandbox/usr</userinput>
                    424: &rprompt; <userinput>cvs -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -d -P pkgsrc</userinput></screen>
                    425: 
                    426:           <para>Do not mount/link this to the copy of your pkgsrc tree
                    427:            you do development in, as this will likely cause problems!
1.21      rillig    428:            </para>
1.1       grant     429:        </step>
                    430: 
                    431:        <step>
                    432:          <para>Make
                    433:            <filename>/usr/sandbox/usr/pkgsrc/packages</filename> and
                    434:            <filename>.../distfiles</filename> point somewhere
                    435:            appropriate. NFS- and/or nullfs-mounts may come in handy!
                    436:            </para>
                    437:        </step>
                    438: 
                    439:        <step>
                    440:          <para>Edit <filename>/etc/mk.conf</filename>, see <xref linkend="binary.mk.conf"/>.</para>
                    441:        </step>
                    442: 
                    443:        <step>
                    444:          <para>Adjust <filename>mk/bulk/build.conf</filename> to suit your needs.</para>
                    445:        </step>
                    446:       </procedure>
                    447: 
1.15      rillig    448:       <para>When the chroot sandbox is set up, you can start
1.1       grant     449:        the build with the following steps:</para>
                    450: 
                    451:       <screen>&rprompt; <userinput>cd /usr/sandbox/usr/pkgsrc</userinput>
                    452: &rprompt; <userinput>sh mk/bulk/do-sandbox-build</userinput></screen>
                    453: 
1.6       hubertf   454:       <para>
                    455:         This will just jump inside the sandbox and start building.  At
                    456:         the end of the build, mail will be sent with the results of
                    457:         the build. Created binary pkgs will be in
                    458:         <filename>/usr/sandbox/usr/pkgsrc/packages</filename>
                    459:         (wherever that points/mounts to/from).</para>
1.1       grant     460:     </sect2>
                    461: 
1.13      wiz       462:     <sect2 id="building-a-partial-set">
1.1       grant     463:       <title>Building a partial set of packages</title>
                    464: 
                    465:       <para> In addition to building a complete set of all packages in
                    466:         pkgsrc, the <filename>pkgsrc/mk/bulk/build</filename> script
                    467:         may be used to build a subset of the packages contained in
1.15      rillig    468:         pkgsrc.  By setting <varname>SPECIFIC_PKGS</varname>
1.1       grant     469:         in <filename>/etc/mk.conf</filename>, the variables</para>
                    470: 
                    471:       <itemizedlist>
                    472:        <listitem><para>SITE_SPECIFIC_PKGS</para></listitem>
                    473:        <listitem><para>HOST_SPECIFIC_PKGS</para></listitem>
                    474:        <listitem><para>GROUP_SPECIFIC_PKGS</para></listitem>
                    475:        <listitem><para>USER_SPECIFIC_PKGS</para></listitem>
                    476:       </itemizedlist>
                    477: 
                    478:       <para> will define the set of packages which should be built.
                    479:         The bulk build code will also include any packages which are
                    480:         needed as dependencies for the explicitly listed packages.
                    481:        </para>
                    482: 
                    483:       <para> One use of this is to do a bulk build with
                    484:         <varname>SPECIFIC_PKGS</varname> in a chroot sandbox
                    485:         periodically to have a complete set of the binary packages
                    486:         needed for your site available without the overhead of
                    487:         building extra packages that are not needed.  </para>
                    488: 
                    489:     </sect2>
1.6       hubertf   490: 
                    491:     <sect2 id="bulk-upload">
                    492:       <title>Uploading results of a bulk build</title>
                    493: 
                    494:       <para>
                    495:         This section describes how pkgsrc developers can upload binary
                    496:         pkgs built by bulk builds to ftp.NetBSD.org.
                    497:       </para>
                    498: 
                    499:       <para>
1.8       jschauma  500:        If you would like to automatically create checksum files for the
                    501:        binary packages you intend to upload, remember to set
                    502:        <varname>MKSUMS=yes</varname> in your
                    503:        <filename>mk/bulk/build.conf</filename>.
                    504:       </para>
                    505: 
                    506:       <para>
                    507:        If you would like to PGP sign the checksum files (highly
                    508:        recommended!), remember to set
                    509:        <varname>SIGN_AS=username@NetBSD.org</varname> in your
                    510:        <filename>mk/bulk/build.conf</filename>.  This will prompt you for
1.9       wiz       511:        your GPG password to sign the files before uploading everything.
1.8       jschauma  512:       </para>
                    513: 
                    514:       <para>
                    515:         Then, make sure that you have <varname>RSYNC_DST</varname>
1.6       hubertf   516:         set properly in your <filename>mk/bulk/build.conf</filename>
                    517:         file, i.e. adjust it to something like one of the following:
                    518:       </para>
                    519: 
1.20      rillig    520:       <screen>RSYNC_DST=ftp.NetBSD.org:/pub/NetBSD/packages/pkgsrc-200xQy/NetBSD-a.b.c/arch/upload </screen>
1.6       hubertf   521: 
1.20      rillig    522:        <para>Please use appropriate values for "pkgsrc-200xQy",
                    523:         "NetBSD-a.b.c" and "arch" here. If your login on ftp.NetBSD.org
                    524:        is different from your local login, write your login directly
                    525:        into the variable, e.g. my local account is "feyrer", but for my
                    526:        login "hubertf", I use:</para>
1.6       hubertf   527: 
                    528:       <screen>RSYNC_DST=hubertf@ftp.NetBSD.org:/pub/NetBSD/packages/pkgsrc-200xQy/NetBSD-a.b.c/arch/upload </screen>
                    529: 
                    530:       <para>
                    531:         A separate <filename>upload</filename> directory is used
                    532:         here to allow "closing" the directory during upload. To do
                    533:         so, run the following command on ftp.NetBSD.org next:
                    534:       </para>
                    535: 
                    536:       <screen>nbftp% <userinput>mkdir -p -m 750 /pub/NetBSD/packages/pkgsrc-200xQy/NetBSD-a.b.c/arch/upload</userinput></screen>
                    537: 
                    538:       <para>
                    539:         Please note that <filename>/pub/NetBSD/packages</filename> is
                    540:         only appropriate for packages for the NetBSD operating
                    541:         system. Binary packages for other operating systems should go
                    542:         into <filename>/pub/pkgsrc</filename>.
                    543:       </para>
                    544: 
                    545:       <para>
                    546:         Before uploading the binary pkgs, ssh authentication needs to
1.15      rillig    547:         be set up. This example shows how to set up temporary keys
1.6       hubertf   548:         for the root account <emphasis>inside the sandbox</emphasis>
                    549:         (assuming that no keys should be present there usually):
                    550:       </para>
                    551: 
1.21      rillig    552:       <screen>&rprompt; <userinput>chroot /usr/sandbox</userinput>
                    553: chroot-&rprompt; <userinput>rm $HOME/.ssh/id-dsa*</userinput>
                    554: chroot-&rprompt; <userinput>ssh-keygen -t dsa</userinput>
1.6       hubertf   555: chroot-&rprompt; <userinput>cat $HOME/.ssh/id-dsa.pub</userinput> </screen>
                    556: 
                    557:       <para>
                    558:         Now take the output of <filename>id-dsa.pub</filename> and
                    559:         append it to your <filename>~/.ssh/authorized_keys</filename>
1.7       heinz     560:         file on ftp.NetBSD.org. You can remove the key after the
1.6       hubertf   561:         upload is done!
                    562:       </para>
                    563: 
                    564:       <para>
                    565:         Next, test if your ssh connection really works:
                    566:       </para>
                    567: 
                    568:       <screen>chroot-&rprompt; <userinput>ssh ftp.NetBSD.org date</userinput> </screen>
                    569: 
                    570:       <para>
                    571:         Use "-l yourNetBSDlogin" here as appropriate!
                    572:       </para>
                    573: 
                    574:       <para>
                    575:         Now after all this works, you can exit the sandbox and start
                    576:         the upload:
                    577:       </para>
                    578: 
1.21      rillig    579:       <screen>chroot-&rprompt; <userinput>exit</userinput>
                    580: &rprompt; <userinput>cd /usr/sandbox/usr/pkgsrc</userinput>
1.6       hubertf   581: &rprompt; <userinput>sh mk/bulk/do-sandbox-upload</userinput> </screen>
                    582: 
                    583:       <para>
1.11      wiz       584:         The upload process may take quite some time. Use &man.ls.1; or
                    585:         &man.du.1; on the FTP server to monitor progress of the
                    586:         upload. The upload script will take care of not uploading
                    587:         restricted packages and putting vulnerable packages into the
                    588:         <filename>vulnerable</filename> subdirectory.
1.6       hubertf   589:       </para>
                    590: 
                    591:       <para>
                    592:         After the upload has ended, first thing is to revoke ssh access:
                    593:       </para>
                    594: 
                    595:       <screen>nbftp% <userinput>vi ~/.ssh/authorized_keys</userinput>
                    596: Gdd:x! </screen>
                    597: 
                    598:       <para>
                    599:         Use whatever is needed to remove the key you've entered
                    600:         before! Last, move the uploaded packages out of the
                    601:         <filename>upload</filename> directory to have them accessible
1.21      rillig    602:         to everyone:
1.6       hubertf   603:       </para>
                    604: 
                    605:       <screen>nbftp% <userinput>cd /pub/NetBSD/packages/pkgsrc-200xQy/NetBSD-a.b.c/arch</userinput>
                    606: nbftp% <userinput>mv upload/* .</userinput>
                    607: nbftp% <userinput>rmdir upload</userinput>
                    608: nbftp% <userinput>chmod 755 .</userinput> </screen>
                    609:     </sect2>
1.1       grant     610:   </sect1>
                    611: 
1.13      wiz       612:   <sect1 id="creating-cdroms">
1.1       grant     613:     <title>Creating a multiple CD-ROM packages collection</title>
                    614: 
1.6       hubertf   615:     <para>
                    616:       After your pkgsrc bulk-build has completed, you may wish to
                    617:       create a CD-ROM set of the resulting binary packages to assist
                    618:       in installing packages on other machines.  The
1.21      rillig    619:       <filename role="pkg">pkgtools/cdpack</filename> package provides
                    620:       a simple tool for creating the ISO 9660 images.
                    621:       <command>cdpack</command> arranges the packages on the CD-ROMs in a
                    622:       way that keeps all the dependencies for a given package on the same
1.14      rpaulo    623:       CD as that package.
1.6       hubertf   624:     </para>
1.1       grant     625: 
1.13      wiz       626:     <sect2 id="cdpack-example">
1.1       grant     627:       <title>Example of cdpack</title>
                    628: 
1.6       hubertf   629:       <para>
                    630:         Complete documentation for cdpack is found in the cdpack(1)
1.15      rillig    631:         man page. The following short example assumes that the binary
1.6       hubertf   632:         packages are left in
                    633:         <filename>/usr/pkgsrc/packages/All</filename> and that
                    634:         sufficient disk space exists in <filename>/u2</filename> to
                    635:         hold the ISO 9660 images.</para>
1.1       grant     636: 
                    637:       <screen>&rprompt; <userinput>mkdir /u2/images</userinput>
                    638: &rprompt; <userinput>pkg_add /usr/pkgsrc/packages/All/cdpack</userinput>
                    639: &rprompt; <userinput>cdpack /usr/pkgsrc/packages/All /u2/images</userinput></screen>
                    640: 
1.6       hubertf   641:       <para>
                    642:         If you wish to include a common set of files
                    643:        (<filename>COPYRIGHT</filename>, <filename>README</filename>,
                    644:        etc.) on each CD in the collection, then you need to create a
                    645:        directory which contains these files. e.g.
                    646:       </para>
1.1       grant     647: 
                    648:       <screen>&rprompt; <userinput>mkdir /tmp/common</userinput>
                    649: &rprompt; <userinput>echo "This is a README" &gt; /tmp/common/README</userinput>
                    650: &rprompt; <userinput>echo "Another file" &gt; /tmp/common/COPYING</userinput>
                    651: &rprompt; <userinput>mkdir /tmp/common/bin</userinput>
                    652: &rprompt; <userinput>echo "#!/bin/sh" &gt; /tmp/common/bin/myscript</userinput>
                    653: &rprompt; <userinput>echo "echo Hello world" &gt;&gt; /tmp/common/bin/myscript</userinput>
                    654: &rprompt; <userinput>chmod 755 /tmp/common/bin/myscript</userinput></screen>
                    655: 
                    656:       <para>Now create the images:</para>
                    657: 
                    658:       <screen>&rprompt; <userinput>cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images</userinput></screen>
                    659: 
                    660:       <para>Each image will contain <filename>README</filename>,
                    661:        <filename>COPYING</filename>, and <filename>bin/myscript</filename>
                    662:        in their root directories.</para>
                    663:     </sect2>
                    664:   </sect1>
                    665: </chapter>

CVSweb <webmaster@jp.NetBSD.org>