Annotation of pkgsrc/doc/guide/files/binary.xml, revision 1.3
1.3 ! agc 1: <!-- $NetBSD: binary.xml,v 1.2 2004/12/23 18:57:03 ben Exp $ -->
1.1 grant 2:
3: <chapter id="binary">
4: <title>Creating binary packages</title>
5:
6: <sect1>
7: <title>Building a single binary package</title>
8:
9: <para>Once you have built and installed a package, you can create a
10: <emphasis>binary package</emphasis> which can be installed on another
11: system with &man.pkg.add.1; This saves having to build the same package on
12: a group of hosts and wasting CPU time. It also provides a simple means
13: for others to install your package, should you distribute it.</para>
14:
15: <para>To create a binary package, change into the appropriate
16: directory in pkgsrc, and run <command>make package</command>:</para>
17:
18: <screen>&rprompt; <userinput>cd misc/figlet</userinput>
19: &rprompt; <userinput>make package</userinput></screen>
20:
21: <para>This will build and install your package (if not already done), and then
22: build a binary package from what was installed. You can then use the
23: <command>pkg_*</command> tools to manipulate it. Binary packages are
24: created by default in <filename>/usr/pkgsrc/packages</filename>, in the
25: form of a gzipped tar file. See <xref linkend="logs.package"/> for
26: a continuation of the above <pkg>misc/figlet</pkg> example.</para>
27:
28: <para>See <xref linkend="submit"/> for information on how to submit such a
29: binary package.</para>
30: </sect1>
31:
32: <sect1>
33: <title>Settings for creation of binary packages</title>
34:
35: <para>See <xref linkend="build.helpful-targets"/>.</para>
36: </sect1>
37:
38: <sect1 id="bulkbuild">
39: <title>Doing a bulk build of all packages</title>
40:
41: <para>If you want to get a full set of precompiled binary packages, this section
42: describes how to get them. Beware that the bulk build will remove all
43: currently installed packages from your system! Having a FTP server
44: configured either on the machine doing the bulk builds or on a nearby NFS
45: server can help to make the packages available to everyone. See
46: &man.ftpd.8; for
47: more information. If you use a remote NFS server's storage, be sure to not
48: actually compile on NFS storage, as this slows things down a lot.</para>
49:
50: <sect2 id="binary.configuration">
51: <title>Configuration</title>
52:
53: <sect3 id="binary.mk.conf">
54: <title>/etc/mk.conf</title>
55:
56: <para>You may want to set things in <filename>/etc/mk.conf</filename>.
1.2 ben 57: Look at <filename>pkgsrc/mk/defaults/mk.conf</filename> for
1.1 grant 58: details of the default settings. You will want to ensure that
59: <varname>ACCEPTABLE_LICENSES</varname> meet your local policy.
60: As used in this example, <varname>_ACCEPTABLE=yes</varname>
61: accepts <emphasis>all</emphasis> licenses.</para>
62:
63: <programlisting>PACKAGES?= ${_PKGSRCDIR}/packages/${MACHINE_ARCH}
64: WRKOBJDIR?= /usr/tmp/pkgsrc # build here instead of in pkgsrc
65: BSDSRCDIR= /usr/src
66: BSDXSRCDIR= /usr/xsrc # for x11/xservers
67: OBJHOSTNAME?= yes # use work.`hostname`
68: FAILOVER_FETCH= yes # insist on the correct checksum
69: PKG_DEVELOPER?= yes
70: _ACCEPTABLE= yes</programlisting>
71: </sect3>
72:
73: <sect3>
74: <title><filename>build.conf</filename></title>
75:
76: <para>In <filename>pkgsrc/mk/bulk</filename>, copy
77: <filename>build.conf-example</filename> to
78: <filename>build.conf</filename> and
79: edit it, following the comments in that file. This is the config
80: file that determines where log files are generated after the build,
81: where to mail the build report to, where your pkgsrc tree is located and
82: which user to &man.su.8; to to do a <command>cvs update</command>.</para>
83: </sect3>
84:
85: <sect3>
86: <title><filename>pre-build.local</filename></title>
87:
88: <para>It is possible to configure the bulk build to perform certain site
89: specific tasks at the end of the pre-build stage. If the file
90: <filename>pre-build.local</filename> exists in
91: <filename>/usr/pkgsrc/mk/bulk</filename> it will be executed
92: (as a sh(1) script) at the end of the usual pre-build stage. An
93: example use of <filename>pre-build.local</filename> is to have the
94: line:</para>
95:
96: <screen>&rprompt; <userinput>echo "I do not have enough disk space to build this pig." \
97: > pkgsrc/games/crafty-book-enormous/$BROKENF</userinput></screen>
98:
99: <para>to prevent the system from trying to build a particular package
100: which requires nearly 3 GB of disk space.</para>
101: </sect3>
102: </sect2>
103:
104: <sect2>
105: <title>Other environmental considerations</title>
106:
107: <para>As <filename>/usr/pkg</filename> will be completely deleted at the
108: start of bulk builds, make sure your login shell is placed somewhere
109: else. Either drop it into <filename>/usr/local/bin</filename>
110: (and adjust your login shell in the passwd file), or (re-)install
111: it via &man.pkg.add.1; from
112: <filename>/etc/rc.local</filename>, so you can login after a reboot
113: (remember that your current process won't die if the package is
114: removed, you just can't start any new instances of the shell any more).
115: Also, if you use &os; earlier than 1.5, or you still want to use the
116: pkgsrc version of ssh for some reason, be sure to install ssh before
117: starting it from <filename>rc.local</filename>:</para>
118:
119: <programlisting>( cd /usr/pkgsrc/security/ssh ; make bulk-install )
120: if [ -f /usr/pkg/etc/rc.d/sshd ]; then
121: /usr/pkg/etc/rc.d/sshd
122: fi</programlisting>
123:
124: <para>Not doing so will result in you being not able to log in via ssh
125: after the bulk build is finished or if the machine gets rebooted
126: or crashes. You have been warned! :)</para>
127: </sect2>
128:
129: <sect2>
130: <title>Operation</title>
131:
132: <para>Make sure you don't need any of the packages still installed.</para>
133:
134: <warning>
135: <para>During the bulk build, <emphasis>all packages will be
136: removed!</emphasis></para>
137: </warning>
138:
139: <para>Be sure to remove all other things that might
140: interfere with builds, like some libs installed in
141: <filename>/usr/local</filename>, etc. then become root and type:</para>
142:
143: <screen>&rprompt; <userinput>cd /usr/pkgsrc</userinput>
144: &rprompt; <userinput>sh mk/bulk/build</userinput></screen>
145:
146: <para>If for some reason your last build didn't complete (power failure,
147: system panic, ...), you can continue it by running:</para>
148:
149: <screen>&rprompt; <userinput>sh mk/bulk/build restart</userinput></screen>
150:
151: <para>At the end of the bulk build, you will get a summary via mail, and find
152: build logs in the directory specified by <varname>FTP</varname> in the
153: <filename>build.conf</filename> file.</para>
154: </sect2>
155:
156: <sect2>
157: <title>What it does</title>
158:
159: <para>The bulk builds consist of three steps:</para>
160:
161: <variablelist>
162: <varlistentry>
163: <term>1. pre-build</term>
164:
165: <listitem>
166: <para>The script updates your pkgsrc tree via (anon)cvs, then cleans
167: out any broken distfiles, and removes all packages installed.</para>
168: </listitem>
169: </varlistentry>
170:
171: <varlistentry>
172: <term>2. the bulk build</term>
173:
174: <listitem>
175: <para>This is basically
176: <quote>make bulk-package</quote> with an optimised
177: order in which packages will be built. Packages that don't require
178: other packages will be built first, and packages with many dependencies
179: will be built later.</para>
180: </listitem>
181: </varlistentry>
182:
183: <varlistentry>
184: <term>3. post-build</term>
185:
186: <listitem>
187: <para>Generates a report that's placed in the directory specified
188: in the <filename>build.conf</filename> file named
189: <filename>broken.html</filename>, a short version of
190: that report will also be mailed to the build's admin.</para>
191: </listitem>
192: </varlistentry>
193: </variablelist>
194:
195: <para>During the build, a list of broken packages will be compiled in
196: <filename>/usr/pkgsrc/.broken</filename> (or
197: <filename>.../.broken.${MACHINE}</filename> if
198: <varname>OBJMACHINE</varname> is set),
199: individual build logs of broken builds can be found in the package's
200: directory. These files are used by the bulk-targets to mark broken builds
201: to not waste time trying to rebuild them, and they can be used to debug
202: these broken package builds later.</para>
203: </sect2>
204:
205: <sect2>
206: <title>Disk space requirements</title>
207:
208: <para>Currently, roughly the following requirements are valid for
209: NetBSD 2.0/i386:</para>
210:
211: <itemizedlist>
212: <listitem>
213: <para>10 GB - distfiles (NFS ok)</para>
214: </listitem>
215:
216: <listitem>
217: <para>8 GB - full set of all binaries (NFS ok)</para>
218: </listitem>
219:
220: <listitem>
221: <para>5 GB - temp space for compiling (local disk recommended)</para>
222: </listitem>
223: </itemizedlist>
224:
225: <para>Note that all pkgs will be de-installed as soon as they are turned into a
226: binary package, and that sources are removed, so there is no excessively huge
227: demand to disk space. Afterwards, if the package is needed again, it will
228: be installed via &man.pkg.add.1; instead of building again, so
229: there are no cycles wasted by recompiling.</para>
230: </sect2>
231:
232: <sect2>
233: <title>Setting up a sandbox for chroot'ed builds</title>
234:
1.3 ! agc 235: <para>If you don't want all the packages nuked from a machine (rendering it useless
1.1 grant 236: for anything but pkg compiling), there is the possibility of doing the pkg
237: bulk build inside a chroot environment.</para>
238:
1.3 ! agc 239: <para>The first step is to set up a chroot
1.1 grant 240: sandbox, e.g. <filename>/usr/sandbox</filename>.
1.3 ! agc 241: This can be done by using null mounts, or manually.</para>
! 242:
! 243: <para>There is a shell script
! 244: called <filename>pkgsrc/bulk/mksandbox</filename> which will set up the sandbox
! 245: environment using null mounts. It will also create a script
! 246: called <filename>sandbox</filename> in the root of the sandbox
! 247: environment, which will allow the null mounts to be activated
! 248: using the <command>sandbox mount</command> and deactivated using
! 249: the <command>sandbox umount</command>.</para>
! 250:
! 251: <para>To set up a sandbox environment by hand,
! 252: after extracting all the sets from a &os; installation or doing a
1.1 grant 253: <command>make distribution DESTDIR=/usr/sandbox</command> in
254: <filename>/usr/src/etc</filename>, be sure the following
255: items are present and properly configured:</para>
256:
257: <procedure>
258: <step>
259: <para>Kernel</para>
260:
261: <screen>&rprompt; <userinput>cp /netbsd /usr/sandbox</userinput></screen>
262: </step>
263:
264: <step>
265: <para><filename>/dev/*</filename></para>
266:
267: <screen>&rprompt; <userinput>cd /usr/sandbox/dev ; sh MAKEDEV all</userinput></screen>
268: </step>
269:
270: <step>
271: <para><filename>/etc/resolv.conf</filename> (for <pkg>security/smtpd</pkg> and mail):</para>
272:
273: <screen>&rprompt; <userinput>cp /etc/resolv.conf /usr/sandbox/etc</userinput></screen>
274: </step>
275:
276: <step>
277: <para>Working(!) mail config (hostname, sendmail.cf):</para>
278:
279: <screen>&rprompt; <userinput>cp /etc/mail/sendmail.cf /usr/sandbox/etc/mail</userinput></screen>
280: </step>
281:
282: <step>
283: <para><filename>/etc/localtime</filename> (for <pkg>security/smtpd</pkg>):</para>
284:
285: <screen>&rprompt; <userinput>ln -sf /usr/share/zoneinfo/UTC /usr/sandbox/etc/localtime</userinput></screen>
286: </step>
287:
288: <step>
289: <para><filename>/usr/src</filename> (system sources, for <pkg>sysutils/aperture</pkg>,
290: <pkg>net/ppp-mppe</pkg>):</para>
291:
292: <screen>&rprompt; <userinput>ln -s ../disk1/cvs .</userinput>
293: &rprompt; <userinput>ln -s cvs/src-1.6 src</userinput></screen>
294: </step>
295:
296: <step>
297: <para>Create <filename>/var/db/pkg</filename> (not part of default install):</para>
298:
299: <screen>&rprompt; <userinput>mkdir /usr/sandbox/var/db/pkg</userinput></screen>
300: </step>
301:
302: <step>
303: <para>Create <filename>/usr/pkg</filename> (not part of default install):</para>
304:
305: <screen>&rprompt; <userinput>mkdir /usr/sandbox/usr/pkg</userinput></screen>
306: </step>
307:
308: <step>
309: <para>Checkout pkgsrc via cvs into
310: <filename>/usr/sandbox/usr/pkgsrc</filename>:</para>
311:
312: <screen>&rprompt; <userinput>cd /usr/sandbox/usr</userinput>
313: &rprompt; <userinput>cvs -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -d -P pkgsrc</userinput></screen>
314:
315: <para>Do not mount/link this to the copy of your pkgsrc tree
316: you do development in, as this will likely cause problems!
317: </para>
318: </step>
319:
320: <step>
321: <para>Make
322: <filename>/usr/sandbox/usr/pkgsrc/packages</filename> and
323: <filename>.../distfiles</filename> point somewhere
324: appropriate. NFS- and/or nullfs-mounts may come in handy!
325: </para>
326: </step>
327:
328: <step>
329: <para>Edit <filename>/etc/mk.conf</filename>, see <xref linkend="binary.mk.conf"/>.</para>
330: </step>
331:
332: <step>
333: <para>Adjust <filename>mk/bulk/build.conf</filename> to suit your needs.</para>
334: </step>
335:
336: <step>
337: <para>If you have set <varname>CVS_USER</varname> in
338: <filename>build.conf</filename>, make sure that
339: account exists and can do a <command>cvs ${CVS_FLAGS}
340: update</command> properly!</para>
341: </step>
342: </procedure>
343:
344: <para>When the chroot sandbox is setup, you can start
345: the build with the following steps:</para>
346:
347: <screen>&rprompt; <userinput>cd /usr/sandbox/usr/pkgsrc</userinput>
348: &rprompt; <userinput>sh mk/bulk/do-sandbox-build</userinput></screen>
349:
350: <para>This will just jump inside the sandbox and start building.
351: At the end of the build, mail will be sent with the results of the build.
352: Created binary pkgs will be in
353: <filename>/usr/sandbox/usr/pkgsrc/packages</filename> (wherever
354: that points/mounts to/from).</para>
355: </sect2>
356:
357: <sect2>
358: <title>Building a partial set of packages</title>
359:
360: <para> In addition to building a complete set of all packages in
361: pkgsrc, the <filename>pkgsrc/mk/bulk/build</filename> script
362: may be used to build a subset of the packages contained in
363: pkgsrc. By setting defining <varname>SPECIFIC_PKGS</varname>
364: in <filename>/etc/mk.conf</filename>, the variables</para>
365:
366: <itemizedlist>
367: <listitem><para>SITE_SPECIFIC_PKGS</para></listitem>
368: <listitem><para>HOST_SPECIFIC_PKGS</para></listitem>
369: <listitem><para>GROUP_SPECIFIC_PKGS</para></listitem>
370: <listitem><para>USER_SPECIFIC_PKGS</para></listitem>
371: </itemizedlist>
372:
373: <para> will define the set of packages which should be built.
374: The bulk build code will also include any packages which are
375: needed as dependencies for the explicitly listed packages.
376: </para>
377:
378: <para> One use of this is to do a bulk build with
379: <varname>SPECIFIC_PKGS</varname> in a chroot sandbox
380: periodically to have a complete set of the binary packages
381: needed for your site available without the overhead of
382: building extra packages that are not needed. </para>
383:
384: </sect2>
385: </sect1>
386:
387: <sect1>
388: <title>Creating a multiple CD-ROM packages collection</title>
389:
390: <para>After your pkgsrc bulk-build has completed, you may wish to create a CD-ROM
391: set of the resulting binary packages to assist in installing packages on
392: other machines. The <pkg>pkgtools/cdpack</pkg> package provides a simple
393: tool for creating the ISO 9660 images. <command>cdpack</command> arranges
394: the packages on the CD-ROMs in a way that keeps all the dependencies for
395: given package on the same CD as that package.</para>
396:
397: <sect2>
398: <title>Example of cdpack</title>
399:
400: <para>Complete documentation for cdpack is found in the
401: cdpack(1) manpage. The following
402: short example assumes that the binary packages are left in
403: <filename>/usr/pkgsrc/packages/All</filename> and that sufficient disk
404: space exists in <filename>/u2</filename> to hold the ISO 9660 images.</para>
405:
406: <screen>&rprompt; <userinput>mkdir /u2/images</userinput>
407: &rprompt; <userinput>pkg_add /usr/pkgsrc/packages/All/cdpack</userinput>
408: &rprompt; <userinput>cdpack /usr/pkgsrc/packages/All /u2/images</userinput></screen>
409:
410: <para>If you wish to include a common set of files
411: (<filename>COPYRIGHT</filename>, <filename>README</filename>, etc.)
412: on each CD in the collection, then you need to create a directory which
413: contains these files. e.g.</para>
414:
415: <screen>&rprompt; <userinput>mkdir /tmp/common</userinput>
416: &rprompt; <userinput>echo "This is a README" > /tmp/common/README</userinput>
417: &rprompt; <userinput>echo "Another file" > /tmp/common/COPYING</userinput>
418: &rprompt; <userinput>mkdir /tmp/common/bin</userinput>
419: &rprompt; <userinput>echo "#!/bin/sh" > /tmp/common/bin/myscript</userinput>
420: &rprompt; <userinput>echo "echo Hello world" >> /tmp/common/bin/myscript</userinput>
421: &rprompt; <userinput>chmod 755 /tmp/common/bin/myscript</userinput></screen>
422:
423: <para>Now create the images:</para>
424:
425: <screen>&rprompt; <userinput>cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images</userinput></screen>
426:
427: <para>Each image will contain <filename>README</filename>,
428: <filename>COPYING</filename>, and <filename>bin/myscript</filename>
429: in their root directories.</para>
430: </sect2>
431: </sect1>
432: </chapter>
CVSweb <webmaster@jp.NetBSD.org>