Annotation of pkgsrc/doc/guide/files/binary.xml, revision 1.8
1.8 ! jschauma 1: <!-- $NetBSD: binary.xml,v 1.7 2005/01/18 23:25:04 heinz Exp $ -->
1.1 grant 2:
3: <chapter id="binary">
4: <title>Creating binary packages</title>
5:
6: <sect1>
7: <title>Building a single binary package</title>
8:
1.6 hubertf 9: <para>
10: Once you have built and installed a package, you can create a
11: <emphasis>binary package</emphasis> which can be installed on
12: another system with &man.pkg.add.1; This saves having to build
13: the same package on a group of hosts and wasting CPU time. It
14: also provides a simple means for others to install your package,
15: should you distribute it.
16: </para>
17:
18: <para>
19: To create a binary package, change into the appropriate
20: directory in pkgsrc, and run <command>make
21: package</command>:
22: </para>
1.1 grant 23:
24: <screen>&rprompt; <userinput>cd misc/figlet</userinput>
25: &rprompt; <userinput>make package</userinput></screen>
26:
1.6 hubertf 27: <para>
28: This will build and install your package (if not already done),
29: and then build a binary package from what was installed. You can
30: then use the <command>pkg_*</command> tools to manipulate
31: it. Binary packages are created by default in
32: <filename>/usr/pkgsrc/packages</filename>, in the form of a
33: gzipped tar file. See <xref linkend="logs.package"/> for a
34: continuation of the above <pkg>misc/figlet</pkg> example.</para>
35:
36: <para>
37: See <xref linkend="submit"/> for information on how to submit
38: such a binary package.</para>
1.1 grant 39: </sect1>
40:
41: <sect1>
42: <title>Settings for creation of binary packages</title>
43:
44: <para>See <xref linkend="build.helpful-targets"/>.</para>
45: </sect1>
46:
47: <sect1 id="bulkbuild">
48: <title>Doing a bulk build of all packages</title>
49:
50: <para>If you want to get a full set of precompiled binary packages, this section
51: describes how to get them. Beware that the bulk build will remove all
52: currently installed packages from your system! Having a FTP server
53: configured either on the machine doing the bulk builds or on a nearby NFS
54: server can help to make the packages available to everyone. See
55: &man.ftpd.8; for
56: more information. If you use a remote NFS server's storage, be sure to not
57: actually compile on NFS storage, as this slows things down a lot.</para>
58:
59: <sect2 id="binary.configuration">
60: <title>Configuration</title>
61:
62: <sect3 id="binary.mk.conf">
63: <title>/etc/mk.conf</title>
64:
65: <para>You may want to set things in <filename>/etc/mk.conf</filename>.
1.2 ben 66: Look at <filename>pkgsrc/mk/defaults/mk.conf</filename> for
1.1 grant 67: details of the default settings. You will want to ensure that
68: <varname>ACCEPTABLE_LICENSES</varname> meet your local policy.
69: As used in this example, <varname>_ACCEPTABLE=yes</varname>
70: accepts <emphasis>all</emphasis> licenses.</para>
71:
72: <programlisting>PACKAGES?= ${_PKGSRCDIR}/packages/${MACHINE_ARCH}
73: WRKOBJDIR?= /usr/tmp/pkgsrc # build here instead of in pkgsrc
74: BSDSRCDIR= /usr/src
75: BSDXSRCDIR= /usr/xsrc # for x11/xservers
76: OBJHOSTNAME?= yes # use work.`hostname`
77: FAILOVER_FETCH= yes # insist on the correct checksum
78: PKG_DEVELOPER?= yes
79: _ACCEPTABLE= yes</programlisting>
80: </sect3>
81:
82: <sect3>
83: <title><filename>build.conf</filename></title>
84:
85: <para>In <filename>pkgsrc/mk/bulk</filename>, copy
86: <filename>build.conf-example</filename> to
87: <filename>build.conf</filename> and
88: edit it, following the comments in that file. This is the config
89: file that determines where log files are generated after the build,
90: where to mail the build report to, where your pkgsrc tree is located and
91: which user to &man.su.8; to to do a <command>cvs update</command>.</para>
92: </sect3>
93:
94: <sect3>
95: <title><filename>pre-build.local</filename></title>
96:
97: <para>It is possible to configure the bulk build to perform certain site
98: specific tasks at the end of the pre-build stage. If the file
99: <filename>pre-build.local</filename> exists in
100: <filename>/usr/pkgsrc/mk/bulk</filename> it will be executed
101: (as a sh(1) script) at the end of the usual pre-build stage. An
102: example use of <filename>pre-build.local</filename> is to have the
103: line:</para>
104:
105: <screen>&rprompt; <userinput>echo "I do not have enough disk space to build this pig." \
106: > pkgsrc/games/crafty-book-enormous/$BROKENF</userinput></screen>
107:
108: <para>to prevent the system from trying to build a particular package
109: which requires nearly 3 GB of disk space.</para>
110: </sect3>
111: </sect2>
112:
113: <sect2>
114: <title>Other environmental considerations</title>
115:
1.6 hubertf 116: <para>As <filename>/usr/pkg</filename> will be completely
117: deleted at the start of bulk builds, make sure your login
118: shell is placed somewhere else. Either drop it into
119: <filename>/usr/local/bin</filename> (and adjust your login
120: shell in the passwd file), or (re-)install it via
121: &man.pkg.add.1; from <filename>/etc/rc.local</filename>, so
122: you can login after a reboot (remember that your current
123: process won't die if the package is removed, you just can't
124: start any new instances of the shell any more). Also, if you
125: use &os; earlier than 1.5, or you still want to use the pkgsrc
126: version of ssh for some reason, be sure to install ssh before
1.1 grant 127: starting it from <filename>rc.local</filename>:</para>
128:
129: <programlisting>( cd /usr/pkgsrc/security/ssh ; make bulk-install )
130: if [ -f /usr/pkg/etc/rc.d/sshd ]; then
131: /usr/pkg/etc/rc.d/sshd
132: fi</programlisting>
133:
1.6 hubertf 134: <para>Not doing so will result in you being not able to log in
135: via ssh after the bulk build is finished or if the machine
136: gets rebooted or crashes. You have been warned! :)</para>
1.1 grant 137: </sect2>
138:
139: <sect2>
140: <title>Operation</title>
141:
1.6 hubertf 142: <para>Make sure you don't need any of the packages still
143: installed.
144: </para>
1.1 grant 145:
146: <warning>
1.6 hubertf 147: <para>During the bulk build, <emphasis>all packages will be
1.1 grant 148: removed!</emphasis></para>
149: </warning>
150:
151: <para>Be sure to remove all other things that might
152: interfere with builds, like some libs installed in
153: <filename>/usr/local</filename>, etc. then become root and type:</para>
154:
155: <screen>&rprompt; <userinput>cd /usr/pkgsrc</userinput>
156: &rprompt; <userinput>sh mk/bulk/build</userinput></screen>
157:
1.6 hubertf 158: <para>If for some reason your last build didn't complete (power
159: failure, system panic, ...), you can continue it by
160: running:</para>
1.1 grant 161:
162: <screen>&rprompt; <userinput>sh mk/bulk/build restart</userinput></screen>
163:
1.6 hubertf 164: <para>
165: At the end of the bulk build, you will get a summary via mail,
166: and find build logs in the directory specified by
167: <varname>FTP</varname> in the <filename>build.conf</filename>
168: file.
169: </para>
1.1 grant 170: </sect2>
171:
172: <sect2>
173: <title>What it does</title>
174:
175: <para>The bulk builds consist of three steps:</para>
176:
177: <variablelist>
178: <varlistentry>
179: <term>1. pre-build</term>
180:
181: <listitem>
1.6 hubertf 182: <para>
183: The script updates your pkgsrc tree via (anon)cvs, then
184: cleans out any broken distfiles, and removes all
185: packages installed.</para>
1.1 grant 186: </listitem>
187: </varlistentry>
188:
189: <varlistentry>
190: <term>2. the bulk build</term>
191:
192: <listitem>
1.6 hubertf 193: <para>
194: This is basically <quote>make bulk-package</quote> with
195: an optimised order in which packages will be
196: built. Packages that don't require other packages will
197: be built first, and packages with many dependencies will
198: be built later.
199: </para>
1.1 grant 200: </listitem>
201: </varlistentry>
202:
203: <varlistentry>
204: <term>3. post-build</term>
205:
206: <listitem>
1.6 hubertf 207: <para>
208: Generates a report that's placed in the directory
209: specified in the <filename>build.conf</filename> file
210: named <filename>broken.html</filename>, a short version
211: of that report will also be mailed to the build's
212: admin.</para>
1.1 grant 213: </listitem>
214: </varlistentry>
215: </variablelist>
216:
1.6 hubertf 217: <para>
218: During the build, a list of broken packages will be compiled
219: in <filename>/usr/pkgsrc/.broken</filename> (or
220: <filename>.../.broken.${MACHINE}</filename> if
221: <varname>OBJMACHINE</varname> is set), individual build logs
222: of broken builds can be found in the package's
223: directory. These files are used by the bulk-targets to mark
224: broken builds to not waste time trying to rebuild them, and
225: they can be used to debug these broken package builds
226: later.
227: </para>
1.1 grant 228: </sect2>
229:
230: <sect2>
231: <title>Disk space requirements</title>
232:
233: <para>Currently, roughly the following requirements are valid for
234: NetBSD 2.0/i386:</para>
235:
236: <itemizedlist>
237: <listitem>
1.6 hubertf 238: <para>
239: 10 GB - distfiles (NFS ok)
240: </para>
1.1 grant 241: </listitem>
242:
243: <listitem>
1.6 hubertf 244: <para>
245: 8 GB - full set of all binaries (NFS ok)
246: </para>
1.1 grant 247: </listitem>
248:
249: <listitem>
1.6 hubertf 250: <para>
251: 5 GB - temp space for compiling (local disk recommended)
252: </para>
1.1 grant 253: </listitem>
254: </itemizedlist>
255:
1.6 hubertf 256: <para>
257: Note that all pkgs will be de-installed as soon as they are
258: turned into a binary package, and that sources are removed,
259: so there is no excessively huge demand to disk
260: space. Afterwards, if the package is needed again, it will
261: be installed via &man.pkg.add.1; instead of building again, so
262: there are no cycles wasted by recompiling.
263: </para>
1.1 grant 264: </sect2>
265:
266: <sect2>
267: <title>Setting up a sandbox for chroot'ed builds</title>
268:
1.6 hubertf 269: <para>
270: If you don't want all the packages nuked from a machine
271: (rendering it useless for anything but pkg compiling), there
272: is the possibility of doing the pkg bulk build inside a
273: chroot environment.
274: </para>
275:
276: <para>
277: The first step is to set up a chroot sandbox,
278: e.g. <filename>/usr/sandbox</filename>. This can be done by
279: using null mounts, or manually.
280: </para>
281:
282: <para>
283: There is a shell script called
284: <filename>pkgsrc/mk/bulk/mksandbox</filename> which will set
285: up the sandbox environment using null mounts. It will also
286: create a script called <filename>sandbox</filename> in the
287: root of the sandbox environment, which will allow the null
288: mounts to be activated using the <command>sandbox
289: mount</command> command and deactivated using the
290: <command>sandbox umount</command> command.
291: </para>
292:
293: <para>
294: To set up a sandbox environment by hand, after extracting all
295: the sets from a &os; installation or doing a <command>make
296: distribution DESTDIR=/usr/sandbox</command> in
297: <filename>/usr/src/etc</filename>, be sure the following items
298: are present and properly configured:
299: </para>
1.1 grant 300:
301: <procedure>
302: <step>
303: <para>Kernel</para>
304:
305: <screen>&rprompt; <userinput>cp /netbsd /usr/sandbox</userinput></screen>
306: </step>
307:
308: <step>
309: <para><filename>/dev/*</filename></para>
310:
311: <screen>&rprompt; <userinput>cd /usr/sandbox/dev ; sh MAKEDEV all</userinput></screen>
312: </step>
313:
314: <step>
315: <para><filename>/etc/resolv.conf</filename> (for <pkg>security/smtpd</pkg> and mail):</para>
316:
317: <screen>&rprompt; <userinput>cp /etc/resolv.conf /usr/sandbox/etc</userinput></screen>
318: </step>
319:
320: <step>
321: <para>Working(!) mail config (hostname, sendmail.cf):</para>
322:
323: <screen>&rprompt; <userinput>cp /etc/mail/sendmail.cf /usr/sandbox/etc/mail</userinput></screen>
324: </step>
325:
326: <step>
327: <para><filename>/etc/localtime</filename> (for <pkg>security/smtpd</pkg>):</para>
328:
329: <screen>&rprompt; <userinput>ln -sf /usr/share/zoneinfo/UTC /usr/sandbox/etc/localtime</userinput></screen>
330: </step>
331:
332: <step>
333: <para><filename>/usr/src</filename> (system sources, for <pkg>sysutils/aperture</pkg>,
334: <pkg>net/ppp-mppe</pkg>):</para>
335:
336: <screen>&rprompt; <userinput>ln -s ../disk1/cvs .</userinput>
337: &rprompt; <userinput>ln -s cvs/src-1.6 src</userinput></screen>
338: </step>
339:
340: <step>
341: <para>Create <filename>/var/db/pkg</filename> (not part of default install):</para>
342:
343: <screen>&rprompt; <userinput>mkdir /usr/sandbox/var/db/pkg</userinput></screen>
344: </step>
345:
346: <step>
347: <para>Create <filename>/usr/pkg</filename> (not part of default install):</para>
348:
349: <screen>&rprompt; <userinput>mkdir /usr/sandbox/usr/pkg</userinput></screen>
350: </step>
351:
352: <step>
353: <para>Checkout pkgsrc via cvs into
354: <filename>/usr/sandbox/usr/pkgsrc</filename>:</para>
355:
356: <screen>&rprompt; <userinput>cd /usr/sandbox/usr</userinput>
357: &rprompt; <userinput>cvs -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -d -P pkgsrc</userinput></screen>
358:
359: <para>Do not mount/link this to the copy of your pkgsrc tree
360: you do development in, as this will likely cause problems!
361: </para>
362: </step>
363:
364: <step>
365: <para>Make
366: <filename>/usr/sandbox/usr/pkgsrc/packages</filename> and
367: <filename>.../distfiles</filename> point somewhere
368: appropriate. NFS- and/or nullfs-mounts may come in handy!
369: </para>
370: </step>
371:
372: <step>
373: <para>Edit <filename>/etc/mk.conf</filename>, see <xref linkend="binary.mk.conf"/>.</para>
374: </step>
375:
376: <step>
377: <para>Adjust <filename>mk/bulk/build.conf</filename> to suit your needs.</para>
378: </step>
379:
380: <step>
381: <para>If you have set <varname>CVS_USER</varname> in
382: <filename>build.conf</filename>, make sure that
383: account exists and can do a <command>cvs ${CVS_FLAGS}
384: update</command> properly!</para>
385: </step>
386: </procedure>
387:
388: <para>When the chroot sandbox is setup, you can start
389: the build with the following steps:</para>
390:
391: <screen>&rprompt; <userinput>cd /usr/sandbox/usr/pkgsrc</userinput>
392: &rprompt; <userinput>sh mk/bulk/do-sandbox-build</userinput></screen>
393:
1.6 hubertf 394: <para>
395: This will just jump inside the sandbox and start building. At
396: the end of the build, mail will be sent with the results of
397: the build. Created binary pkgs will be in
398: <filename>/usr/sandbox/usr/pkgsrc/packages</filename>
399: (wherever that points/mounts to/from).</para>
1.1 grant 400: </sect2>
401:
402: <sect2>
403: <title>Building a partial set of packages</title>
404:
405: <para> In addition to building a complete set of all packages in
406: pkgsrc, the <filename>pkgsrc/mk/bulk/build</filename> script
407: may be used to build a subset of the packages contained in
408: pkgsrc. By setting defining <varname>SPECIFIC_PKGS</varname>
409: in <filename>/etc/mk.conf</filename>, the variables</para>
410:
411: <itemizedlist>
412: <listitem><para>SITE_SPECIFIC_PKGS</para></listitem>
413: <listitem><para>HOST_SPECIFIC_PKGS</para></listitem>
414: <listitem><para>GROUP_SPECIFIC_PKGS</para></listitem>
415: <listitem><para>USER_SPECIFIC_PKGS</para></listitem>
416: </itemizedlist>
417:
418: <para> will define the set of packages which should be built.
419: The bulk build code will also include any packages which are
420: needed as dependencies for the explicitly listed packages.
421: </para>
422:
423: <para> One use of this is to do a bulk build with
424: <varname>SPECIFIC_PKGS</varname> in a chroot sandbox
425: periodically to have a complete set of the binary packages
426: needed for your site available without the overhead of
427: building extra packages that are not needed. </para>
428:
429: </sect2>
1.6 hubertf 430:
431: <sect2 id="bulk-upload">
432: <title>Uploading results of a bulk build</title>
433:
434: <para>
435: This section describes how pkgsrc developers can upload binary
436: pkgs built by bulk builds to ftp.NetBSD.org.
437: </para>
438:
439: <para>
1.8 ! jschauma 440: If you would like to automatically create checksum files for the
! 441: binary packages you intend to upload, remember to set
! 442: <varname>MKSUMS=yes</varname> in your
! 443: <filename>mk/bulk/build.conf</filename>.
! 444: </para>
! 445:
! 446: <para>
! 447: If you would like to PGP sign the checksum files (highly
! 448: recommended!), remember to set
! 449: <varname>SIGN_AS=username@NetBSD.org</varname> in your
! 450: <filename>mk/bulk/build.conf</filename>. This will prompt you for
! 451: your gpg password to sign the files before uploading everything.
! 452: </para>
! 453:
! 454: <para>
! 455: Then, make sure that you have <varname>RSYNC_DST</varname>
1.6 hubertf 456: set properly in your <filename>mk/bulk/build.conf</filename>
457: file, i.e. adjust it to something like one of the following:
458: </para>
459:
460: <screen>RSYNC_DST=$CVS_USER@ftp.NetBSD.org:/pub/NetBSD/packages/pkgsrc-200xQy/NetBSD-a.b.c/arch/upload </screen>
461:
462: <para>
463: Please use appropviate values for "pkgsrc-200xQ4",
464: "NetBSD-a.b.c" and "arch" here. If your login on
465: ftp.NetBSD.org is different from <varname>CVS_USER</varname>,
466: write your login directly into the variable, e.g. my local
467: account is "feyrer", but for my login "hubertf", I use:
468: </para>
469:
470: <screen>RSYNC_DST=hubertf@ftp.NetBSD.org:/pub/NetBSD/packages/pkgsrc-200xQy/NetBSD-a.b.c/arch/upload </screen>
471:
472: <para>
473: A separate <filename>upload</filename> directory is used
474: here to allow "closing" the directory during upload. To do
475: so, run the following command on ftp.NetBSD.org next:
476: </para>
477:
478: <screen>nbftp% <userinput>mkdir -p -m 750 /pub/NetBSD/packages/pkgsrc-200xQy/NetBSD-a.b.c/arch/upload</userinput></screen>
479:
480: <para>
481: Please note that <filename>/pub/NetBSD/packages</filename> is
482: only appropriate for packages for the NetBSD operating
483: system. Binary packages for other operating systems should go
484: into <filename>/pub/pkgsrc</filename>.
485: </para>
486:
487: <para>
488: Before uploading the binary pkgs, ssh authentication needs to
489: be setup next. This example shows how to setup temporary keys
490: for the root account <emphasis>inside the sandbox</emphasis>
491: (assuming that no keys should be present there usually):
492: </para>
493:
494: <screen>&rprompt; <userinput>chroot /usr/sandbox</userinput>
495: chroot-&rprompt; <userinput>rm $HOME/.ssh/id-dsa*</userinput>
496: chroot-&rprompt; <userinput>ssh-keygen -t dsa</userinput>
497: chroot-&rprompt; <userinput>cat $HOME/.ssh/id-dsa.pub</userinput> </screen>
498:
499: <para>
500: Now take the output of <filename>id-dsa.pub</filename> and
501: append it to your <filename>~/.ssh/authorized_keys</filename>
1.7 heinz 502: file on ftp.NetBSD.org. You can remove the key after the
1.6 hubertf 503: upload is done!
504: </para>
505:
506: <para>
507: Next, test if your ssh connection really works:
508: </para>
509:
510: <screen>chroot-&rprompt; <userinput>ssh ftp.NetBSD.org date</userinput> </screen>
511:
512: <para>
513: Use "-l yourNetBSDlogin" here as appropriate!
514: </para>
515:
516: <para>
517: Now after all this works, you can exit the sandbox and start
518: the upload:
519: </para>
520:
521: <screen>chroot-&rprompt; <userinput>exit</userinput>
522: &rprompt; <userinput>cd /usr/sandbox/usr/pkgsrc</userinput>
523: &rprompt; <userinput>sh mk/bulk/do-sandbox-upload</userinput> </screen>
524:
525: <para>
526: The upload process may take quite some time. Use "ls" or "du"
527: on the FTP server to monitor progress of the upload.
528: </para>
529:
530: <para>
531: After the upload has ended, first thing is to revoke ssh access:
532: </para>
533:
534: <screen>nbftp% <userinput>vi ~/.ssh/authorized_keys</userinput>
535: Gdd:x! </screen>
536:
537: <para>
538: Use whatever is needed to remove the key you've entered
539: before! Last, move the uploaded packages out of the
540: <filename>upload</filename> directory to have them accessible
541: to everyone:
542: </para>
543:
544: <screen>nbftp% <userinput>cd /pub/NetBSD/packages/pkgsrc-200xQy/NetBSD-a.b.c/arch</userinput>
545: nbftp% <userinput>mv upload/* .</userinput>
546: nbftp% <userinput>rmdir upload</userinput>
547: nbftp% <userinput>chmod 755 .</userinput> </screen>
548: </sect2>
1.1 grant 549: </sect1>
550:
551: <sect1>
552: <title>Creating a multiple CD-ROM packages collection</title>
553:
1.6 hubertf 554: <para>
555: After your pkgsrc bulk-build has completed, you may wish to
556: create a CD-ROM set of the resulting binary packages to assist
557: in installing packages on other machines. The
558: <pkg>pkgtools/cdpack</pkg> package provides a simple tool for
559: creating the ISO 9660 images. <command>cdpack</command> arranges
560: the packages on the CD-ROMs in a way that keeps all the
561: dependencies for given package on the same CD as that
562: package.
563: </para>
1.1 grant 564:
565: <sect2>
566: <title>Example of cdpack</title>
567:
1.6 hubertf 568: <para>
569: Complete documentation for cdpack is found in the cdpack(1)
570: manpage. The following short example assumes that the binary
571: packages are left in
572: <filename>/usr/pkgsrc/packages/All</filename> and that
573: sufficient disk space exists in <filename>/u2</filename> to
574: hold the ISO 9660 images.</para>
1.1 grant 575:
576: <screen>&rprompt; <userinput>mkdir /u2/images</userinput>
577: &rprompt; <userinput>pkg_add /usr/pkgsrc/packages/All/cdpack</userinput>
578: &rprompt; <userinput>cdpack /usr/pkgsrc/packages/All /u2/images</userinput></screen>
579:
1.6 hubertf 580: <para>
581: If you wish to include a common set of files
582: (<filename>COPYRIGHT</filename>, <filename>README</filename>,
583: etc.) on each CD in the collection, then you need to create a
584: directory which contains these files. e.g.
585: </para>
1.1 grant 586:
587: <screen>&rprompt; <userinput>mkdir /tmp/common</userinput>
588: &rprompt; <userinput>echo "This is a README" > /tmp/common/README</userinput>
589: &rprompt; <userinput>echo "Another file" > /tmp/common/COPYING</userinput>
590: &rprompt; <userinput>mkdir /tmp/common/bin</userinput>
591: &rprompt; <userinput>echo "#!/bin/sh" > /tmp/common/bin/myscript</userinput>
592: &rprompt; <userinput>echo "echo Hello world" >> /tmp/common/bin/myscript</userinput>
593: &rprompt; <userinput>chmod 755 /tmp/common/bin/myscript</userinput></screen>
594:
595: <para>Now create the images:</para>
596:
597: <screen>&rprompt; <userinput>cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images</userinput></screen>
598:
599: <para>Each image will contain <filename>README</filename>,
600: <filename>COPYING</filename>, and <filename>bin/myscript</filename>
601: in their root directories.</para>
602: </sect2>
603: </sect1>
604: </chapter>
CVSweb <webmaster@jp.NetBSD.org>