Annotation of pkgsrc/doc/guide/files/binary.xml, revision 1.1

1.1     ! grant       1: <!-- $NetBSD: binary.xml,v 1.3 2004/09/29 13:08:36 hubertf Exp $ -->
        !             2: 
        !             3: <chapter id="binary">
        !             4:   <title>Creating binary packages</title>
        !             5: 
        !             6:   <sect1>
        !             7:     <title>Building a single binary package</title>
        !             8: 
        !             9:     <para>Once you have built and installed a package, you can create a
        !            10:       <emphasis>binary package</emphasis> which can be installed on another
        !            11:       system with &man.pkg.add.1;  This saves having to build the same package on
        !            12:       a group of hosts and wasting CPU time. It also provides a simple means
        !            13:       for others to install your package, should you distribute it.</para>
        !            14: 
        !            15:     <para>To create a binary package, change into the appropriate
        !            16:       directory in pkgsrc, and run <command>make package</command>:</para>
        !            17: 
        !            18:     <screen>&rprompt; <userinput>cd misc/figlet</userinput>
        !            19: &rprompt; <userinput>make package</userinput></screen>
        !            20: 
        !            21:     <para>This will build and install your package (if not already done), and then
        !            22:       build a binary package from what was installed. You can then use the
        !            23:       <command>pkg_*</command> tools to manipulate it. Binary packages are
        !            24:       created by default in <filename>/usr/pkgsrc/packages</filename>, in the
        !            25:       form of a gzipped tar file. See <xref linkend="logs.package"/> for
        !            26:       a continuation of the above <pkg>misc/figlet</pkg> example.</para>
        !            27: 
        !            28:     <para>See <xref linkend="submit"/> for information on how to submit such a
        !            29:       binary package.</para>
        !            30:   </sect1>
        !            31: 
        !            32:   <sect1>
        !            33:     <title>Settings for creation of binary packages</title>
        !            34: 
        !            35:     <para>See <xref linkend="build.helpful-targets"/>.</para>
        !            36:   </sect1>
        !            37: 
        !            38:   <sect1 id="bulkbuild">
        !            39:     <title>Doing a bulk build of all packages</title>
        !            40: 
        !            41:     <para>If you want to get a full set of precompiled binary packages, this section
        !            42:       describes how to get them. Beware that the bulk build will remove all
        !            43:       currently installed packages from your system! Having a FTP server
        !            44:       configured either on the machine doing the bulk builds or on a nearby NFS
        !            45:       server can help to make the packages available to everyone. See
        !            46:       &man.ftpd.8; for
        !            47:       more information. If you use a remote NFS server's storage, be sure to not
        !            48:       actually compile on NFS storage, as this slows things down a lot.</para>
        !            49: 
        !            50:     <sect2 id="binary.configuration">
        !            51:       <title>Configuration</title>
        !            52: 
        !            53:       <sect3 id="binary.mk.conf">
        !            54:        <title>/etc/mk.conf</title>
        !            55: 
        !            56:        <para>You may want to set things in <filename>/etc/mk.conf</filename>.
        !            57:          Look at <filename>pkgsrc/mk/bsd.pkg.defaults.mk</filename> for
        !            58:          details of the default settings. You will want to ensure that
        !            59:          <varname>ACCEPTABLE_LICENSES</varname> meet your local policy.
        !            60:          As used in this example, <varname>_ACCEPTABLE=yes</varname>
        !            61:          accepts <emphasis>all</emphasis> licenses.</para>
        !            62: 
        !            63:        <programlisting>PACKAGES?=      ${_PKGSRCDIR}/packages/${MACHINE_ARCH}
        !            64: WRKOBJDIR?=     /usr/tmp/pkgsrc   # build here instead of in pkgsrc
        !            65: BSDSRCDIR=      /usr/src
        !            66: BSDXSRCDIR=     /usr/xsrc         # for x11/xservers
        !            67: OBJHOSTNAME?=   yes               # use work.`hostname`
        !            68: FAILOVER_FETCH= yes               # insist on the correct checksum
        !            69: PKG_DEVELOPER?= yes
        !            70: _ACCEPTABLE=    yes</programlisting>
        !            71:       </sect3>
        !            72: 
        !            73:       <sect3>
        !            74:        <title><filename>build.conf</filename></title>
        !            75: 
        !            76:        <para>In <filename>pkgsrc/mk/bulk</filename>, copy
        !            77:          <filename>build.conf-example</filename> to
        !            78:          <filename>build.conf</filename> and
        !            79:          edit it, following the comments in that file. This is the config
        !            80:          file that determines where log files are generated after the build,
        !            81:          where to mail the build report to, where your pkgsrc tree is located and
        !            82:          which user to &man.su.8; to to do a <command>cvs update</command>.</para>
        !            83:       </sect3>
        !            84: 
        !            85:       <sect3>
        !            86:        <title><filename>pre-build.local</filename></title>
        !            87: 
        !            88:        <para>It is possible to configure the bulk build to perform certain site
        !            89:          specific tasks at the end of the pre-build stage.  If the file
        !            90:          <filename>pre-build.local</filename> exists in
        !            91:          <filename>/usr/pkgsrc/mk/bulk</filename> it will be executed
        !            92:          (as a sh(1) script) at the end of the usual pre-build stage.  An
        !            93:          example use of <filename>pre-build.local</filename> is to have the
        !            94:          line:</para>
        !            95: 
        !            96:        <screen>&rprompt; <userinput>echo "I do not have enough disk space to build this pig." \
        !            97:     &gt; pkgsrc/games/crafty-book-enormous/$BROKENF</userinput></screen>
        !            98: 
        !            99:        <para>to prevent the system from trying to build a particular package
        !           100:          which requires nearly 3 GB of disk space.</para>
        !           101:       </sect3>
        !           102:     </sect2>
        !           103: 
        !           104:     <sect2>
        !           105:       <title>Other environmental considerations</title>
        !           106: 
        !           107:       <para>As <filename>/usr/pkg</filename> will be completely deleted at the
        !           108:        start of bulk builds, make sure your login shell is placed somewhere
        !           109:        else. Either drop it into <filename>/usr/local/bin</filename>
        !           110:        (and adjust your login shell in the passwd file), or (re-)install
        !           111:        it via &man.pkg.add.1; from
        !           112:        <filename>/etc/rc.local</filename>, so you can login after a reboot
        !           113:        (remember that your current process won't die if the package is
        !           114:        removed, you just can't start any new instances of the shell any more).
        !           115:        Also, if you use &os; earlier than 1.5, or you still want to use the
        !           116:        pkgsrc version of ssh for some reason, be sure to install ssh before
        !           117:        starting it from <filename>rc.local</filename>:</para>
        !           118: 
        !           119:       <programlisting>( cd /usr/pkgsrc/security/ssh ; make bulk-install )
        !           120: if [ -f        /usr/pkg/etc/rc.d/sshd ]; then
        !           121:     /usr/pkg/etc/rc.d/sshd
        !           122: fi</programlisting>
        !           123: 
        !           124:       <para>Not doing so will result in you being not able to log in via ssh
        !           125:        after the bulk build is finished or if the machine gets rebooted
        !           126:        or crashes. You have been warned! :)</para>
        !           127:     </sect2>
        !           128: 
        !           129:     <sect2>
        !           130:       <title>Operation</title>
        !           131: 
        !           132:       <para>Make sure you don't need any of the packages still installed.</para>
        !           133: 
        !           134:       <warning>
        !           135:        <para>During the bulk build, <emphasis>all packages will be
        !           136:            removed!</emphasis></para>
        !           137:       </warning>
        !           138: 
        !           139:       <para>Be sure to remove all other things that might
        !           140:        interfere with builds, like some libs installed in
        !           141:        <filename>/usr/local</filename>, etc. then become root and type:</para>
        !           142: 
        !           143:       <screen>&rprompt; <userinput>cd /usr/pkgsrc</userinput>
        !           144: &rprompt; <userinput>sh mk/bulk/build</userinput></screen>
        !           145: 
        !           146:       <para>If for some reason your last build didn't complete (power failure,
        !           147:        system panic, ...), you can continue it by running:</para>
        !           148: 
        !           149:       <screen>&rprompt; <userinput>sh mk/bulk/build restart</userinput></screen>
        !           150: 
        !           151:       <para>At the end of the bulk build, you will get a summary via mail, and find
        !           152:        build logs in the directory specified by <varname>FTP</varname> in the
        !           153:        <filename>build.conf</filename> file.</para>
        !           154:     </sect2>
        !           155: 
        !           156:     <sect2>
        !           157:       <title>What it does</title>
        !           158: 
        !           159:       <para>The bulk builds consist of three steps:</para>
        !           160: 
        !           161:       <variablelist>
        !           162:        <varlistentry>
        !           163:          <term>1. pre-build</term>
        !           164: 
        !           165:          <listitem>
        !           166:            <para>The script updates your pkgsrc tree via (anon)cvs, then cleans
        !           167:              out any broken distfiles, and removes all packages installed.</para>
        !           168:          </listitem>
        !           169:        </varlistentry>
        !           170: 
        !           171:        <varlistentry>
        !           172:          <term>2. the bulk build</term>
        !           173: 
        !           174:          <listitem>
        !           175:            <para>This is basically
        !           176:              <quote>make bulk-package</quote> with an optimised
        !           177:              order in which packages will be built. Packages that don't require
        !           178:              other packages will be built first, and packages with many dependencies
        !           179:              will be built later.</para>
        !           180:          </listitem>
        !           181:        </varlistentry>
        !           182: 
        !           183:        <varlistentry>
        !           184:          <term>3. post-build</term>
        !           185: 
        !           186:          <listitem>
        !           187:            <para>Generates a report that's placed in the directory specified
        !           188:              in the <filename>build.conf</filename> file named
        !           189:              <filename>broken.html</filename>, a short version of
        !           190:              that report will also be mailed to the build's admin.</para>
        !           191:          </listitem>
        !           192:        </varlistentry>
        !           193:       </variablelist>
        !           194: 
        !           195:       <para>During the build, a list of broken packages will be compiled in
        !           196:        <filename>/usr/pkgsrc/.broken</filename> (or
        !           197:        <filename>.../.broken.${MACHINE}</filename> if
        !           198:        <varname>OBJMACHINE</varname> is set),
        !           199:        individual build logs of broken builds can be found in the package's
        !           200:        directory. These files are used by the bulk-targets to mark broken builds
        !           201:        to not waste time trying to rebuild them, and they can be used to debug
        !           202:        these broken package builds later.</para>
        !           203:     </sect2>
        !           204: 
        !           205:     <sect2>
        !           206:       <title>Disk space requirements</title>
        !           207: 
        !           208:       <para>Currently, roughly the following requirements are valid for
        !           209:        NetBSD 2.0/i386:</para>
        !           210: 
        !           211:       <itemizedlist>
        !           212:        <listitem>
        !           213:          <para>10 GB - distfiles (NFS ok)</para>
        !           214:        </listitem>
        !           215: 
        !           216:        <listitem>
        !           217:          <para>8 GB - full set of all binaries (NFS ok)</para>
        !           218:        </listitem>
        !           219: 
        !           220:        <listitem>
        !           221:          <para>5 GB - temp space for compiling (local disk recommended)</para>
        !           222:        </listitem>
        !           223:       </itemizedlist>
        !           224: 
        !           225:       <para>Note that all pkgs will be de-installed as soon as they are turned into a
        !           226:        binary package, and that sources are removed, so there is no excessively huge
        !           227:        demand to disk space. Afterwards, if the package is needed again, it will
        !           228:        be installed via &man.pkg.add.1; instead of building again, so
        !           229:        there are no cycles wasted by recompiling.</para>
        !           230:     </sect2>
        !           231: 
        !           232:     <sect2>
        !           233:       <title>Setting up a sandbox for chroot'ed builds</title>
        !           234: 
        !           235:       <para>If you don't want all the pkgs nuked from a machine (rendering it useless
        !           236:        for anything but pkg compiling), there is the possibility of doing the pkg
        !           237:        bulk build inside a chroot environment.</para>
        !           238: 
        !           239:       <para>The first step to do so is setting up a chroot
        !           240:        sandbox, e.g. <filename>/usr/sandbox</filename>.
        !           241:        After extracting all the sets from a &os; installation or doing a
        !           242:        <command>make distribution DESTDIR=/usr/sandbox</command> in
        !           243:        <filename>/usr/src/etc</filename>, be sure the following
        !           244:        items are present and properly configured:</para>
        !           245: 
        !           246:       <procedure>
        !           247:        <step>
        !           248:          <para>Kernel</para>
        !           249: 
        !           250:          <screen>&rprompt; <userinput>cp /netbsd /usr/sandbox</userinput></screen>
        !           251:        </step>
        !           252: 
        !           253:        <step>
        !           254:          <para><filename>/dev/*</filename></para>
        !           255: 
        !           256:          <screen>&rprompt; <userinput>cd /usr/sandbox/dev ; sh MAKEDEV all</userinput></screen>
        !           257:        </step>
        !           258: 
        !           259:        <step>
        !           260:          <para><filename>/etc/resolv.conf</filename> (for <pkg>security/smtpd</pkg> and mail):</para>
        !           261: 
        !           262:          <screen>&rprompt; <userinput>cp /etc/resolv.conf /usr/sandbox/etc</userinput></screen>
        !           263:        </step>
        !           264: 
        !           265:        <step>
        !           266:          <para>Working(!) mail config (hostname, sendmail.cf):</para>
        !           267: 
        !           268:          <screen>&rprompt; <userinput>cp /etc/mail/sendmail.cf /usr/sandbox/etc/mail</userinput></screen>
        !           269:        </step>
        !           270: 
        !           271:        <step>
        !           272:          <para><filename>/etc/localtime</filename> (for <pkg>security/smtpd</pkg>):</para>
        !           273: 
        !           274:          <screen>&rprompt; <userinput>ln -sf /usr/share/zoneinfo/UTC /usr/sandbox/etc/localtime</userinput></screen>
        !           275:        </step>
        !           276: 
        !           277:        <step>
        !           278:          <para><filename>/usr/src</filename> (system sources, for <pkg>sysutils/aperture</pkg>,
        !           279:            <pkg>net/ppp-mppe</pkg>):</para>
        !           280: 
        !           281:          <screen>&rprompt; <userinput>ln -s ../disk1/cvs .</userinput>
        !           282: &rprompt; <userinput>ln -s cvs/src-1.6 src</userinput></screen>
        !           283:        </step>
        !           284: 
        !           285:        <step>
        !           286:          <para>Create <filename>/var/db/pkg</filename> (not part of default install):</para>
        !           287: 
        !           288:          <screen>&rprompt; <userinput>mkdir /usr/sandbox/var/db/pkg</userinput></screen>
        !           289:        </step>
        !           290: 
        !           291:        <step>
        !           292:          <para>Create <filename>/usr/pkg</filename> (not part of default install):</para>
        !           293: 
        !           294:          <screen>&rprompt; <userinput>mkdir /usr/sandbox/usr/pkg</userinput></screen>
        !           295:        </step>
        !           296: 
        !           297:        <step>
        !           298:          <para>Checkout pkgsrc via cvs into
        !           299:            <filename>/usr/sandbox/usr/pkgsrc</filename>:</para>
        !           300: 
        !           301:          <screen>&rprompt; <userinput>cd /usr/sandbox/usr</userinput>
        !           302: &rprompt; <userinput>cvs -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -d -P pkgsrc</userinput></screen>
        !           303: 
        !           304:           <para>Do not mount/link this to the copy of your pkgsrc tree
        !           305:            you do development in, as this will likely cause problems!
        !           306:            </para> 
        !           307:        </step>
        !           308: 
        !           309:        <step>
        !           310:          <para>Make
        !           311:            <filename>/usr/sandbox/usr/pkgsrc/packages</filename> and
        !           312:            <filename>.../distfiles</filename> point somewhere
        !           313:            appropriate. NFS- and/or nullfs-mounts may come in handy!
        !           314:            </para>
        !           315:        </step>
        !           316: 
        !           317:        <step>
        !           318:          <para>Edit <filename>/etc/mk.conf</filename>, see <xref linkend="binary.mk.conf"/>.</para>
        !           319:        </step>
        !           320: 
        !           321:        <step>
        !           322:          <para>Adjust <filename>mk/bulk/build.conf</filename> to suit your needs.</para>
        !           323:        </step>
        !           324: 
        !           325:        <step>
        !           326:          <para>If you have set <varname>CVS_USER</varname> in
        !           327:            <filename>build.conf</filename>, make sure that 
        !           328:            account exists and can do a <command>cvs ${CVS_FLAGS}
        !           329:            update</command> properly!</para> 
        !           330:        </step>
        !           331:       </procedure>
        !           332: 
        !           333:       <para>When the chroot sandbox is setup, you can start
        !           334:        the build with the following steps:</para>
        !           335: 
        !           336:       <screen>&rprompt; <userinput>cd /usr/sandbox/usr/pkgsrc</userinput>
        !           337: &rprompt; <userinput>sh mk/bulk/do-sandbox-build</userinput></screen>
        !           338: 
        !           339:       <para>This will just jump inside the sandbox and start building.
        !           340:        At the end of the build, mail will be sent with the results of the build.
        !           341:        Created binary pkgs will be in
        !           342:        <filename>/usr/sandbox/usr/pkgsrc/packages</filename> (wherever
        !           343:        that points/mounts to/from).</para>
        !           344:     </sect2>
        !           345: 
        !           346:     <sect2>
        !           347:       <title>Building a partial set of packages</title>
        !           348: 
        !           349:       <para> In addition to building a complete set of all packages in
        !           350:         pkgsrc, the <filename>pkgsrc/mk/bulk/build</filename> script
        !           351:         may be used to build a subset of the packages contained in
        !           352:         pkgsrc.  By setting defining <varname>SPECIFIC_PKGS</varname>
        !           353:         in <filename>/etc/mk.conf</filename>, the variables</para>
        !           354: 
        !           355:       <itemizedlist>
        !           356:        <listitem><para>SITE_SPECIFIC_PKGS</para></listitem>
        !           357:        <listitem><para>HOST_SPECIFIC_PKGS</para></listitem>
        !           358:        <listitem><para>GROUP_SPECIFIC_PKGS</para></listitem>
        !           359:        <listitem><para>USER_SPECIFIC_PKGS</para></listitem>
        !           360:       </itemizedlist>
        !           361: 
        !           362:       <para> will define the set of packages which should be built.
        !           363:         The bulk build code will also include any packages which are
        !           364:         needed as dependencies for the explicitly listed packages.
        !           365:        </para>
        !           366: 
        !           367:       <para> One use of this is to do a bulk build with
        !           368:         <varname>SPECIFIC_PKGS</varname> in a chroot sandbox
        !           369:         periodically to have a complete set of the binary packages
        !           370:         needed for your site available without the overhead of
        !           371:         building extra packages that are not needed.  </para>
        !           372: 
        !           373:     </sect2>
        !           374:   </sect1>
        !           375: 
        !           376:   <sect1>
        !           377:     <title>Creating a multiple CD-ROM packages collection</title>
        !           378: 
        !           379:     <para>After your pkgsrc bulk-build has completed, you may wish to create a CD-ROM
        !           380:       set of the resulting binary packages to assist in installing packages on
        !           381:       other machines.  The <pkg>pkgtools/cdpack</pkg> package provides a simple
        !           382:       tool for creating the ISO 9660 images. <command>cdpack</command> arranges
        !           383:       the packages on the CD-ROMs in a way that keeps all the dependencies for
        !           384:       given package on the same CD as that package.</para>
        !           385: 
        !           386:     <sect2>
        !           387:       <title>Example of cdpack</title>
        !           388: 
        !           389:       <para>Complete documentation for cdpack is found in the
        !           390:        cdpack(1) manpage. The following
        !           391:        short example assumes that the binary packages are left in
        !           392:        <filename>/usr/pkgsrc/packages/All</filename> and that sufficient disk
        !           393:        space exists in <filename>/u2</filename> to hold the ISO 9660 images.</para>
        !           394: 
        !           395:       <screen>&rprompt; <userinput>mkdir /u2/images</userinput>
        !           396: &rprompt; <userinput>pkg_add /usr/pkgsrc/packages/All/cdpack</userinput>
        !           397: &rprompt; <userinput>cdpack /usr/pkgsrc/packages/All /u2/images</userinput></screen>
        !           398: 
        !           399:       <para>If you wish to include a common set of files
        !           400:        (<filename>COPYRIGHT</filename>, <filename>README</filename>, etc.)
        !           401:        on each CD in the collection, then you need to create a directory which
        !           402:        contains these files. e.g.</para>
        !           403: 
        !           404:       <screen>&rprompt; <userinput>mkdir /tmp/common</userinput>
        !           405: &rprompt; <userinput>echo "This is a README" &gt; /tmp/common/README</userinput>
        !           406: &rprompt; <userinput>echo "Another file" &gt; /tmp/common/COPYING</userinput>
        !           407: &rprompt; <userinput>mkdir /tmp/common/bin</userinput>
        !           408: &rprompt; <userinput>echo "#!/bin/sh" &gt; /tmp/common/bin/myscript</userinput>
        !           409: &rprompt; <userinput>echo "echo Hello world" &gt;&gt; /tmp/common/bin/myscript</userinput>
        !           410: &rprompt; <userinput>chmod 755 /tmp/common/bin/myscript</userinput></screen>
        !           411: 
        !           412:       <para>Now create the images:</para>
        !           413: 
        !           414:       <screen>&rprompt; <userinput>cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images</userinput></screen>
        !           415: 
        !           416:       <para>Each image will contain <filename>README</filename>,
        !           417:        <filename>COPYING</filename>, and <filename>bin/myscript</filename>
        !           418:        in their root directories.</para>
        !           419:     </sect2>
        !           420:   </sect1>
        !           421: </chapter>

CVSweb <webmaster@jp.NetBSD.org>