Annotation of htdocs/docs/pkgsrc/bulk.html, revision 1.7
1.1 rillig 1: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
2: <html>
3: <head>
4: <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
5: <title>Chapter 7. Creating binary packages for everything in pkgsrc (bulk builds)</title>
6: <link rel="stylesheet" href="/global.css" type="text/css">
7: <meta name="generator" content="DocBook XSL Stylesheets VX.X.X">
1.6 rillig 8: <link rel="home" href="index.html" title="The pkgsrc guide">
1.1 rillig 9: <link rel="up" href="users-guide.html" title="Part I. The pkgsrc user's guide">
10: <link rel="prev" href="binary.html" title="Chapter 6. Creating binary packages">
11: <link rel="next" href="files.html" title="Chapter 8. Directory layout of the installed files">
12: </head>
13: <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
14: <div class="navheader">
15: <table width="100%" summary="Navigation header">
16: <tr><th colspan="3" align="center">Chapter 7. Creating binary packages for everything in pkgsrc (bulk
17: builds)</th></tr>
18: <tr>
19: <td width="20%" align="left">
20: <a accesskey="p" href="binary.html">Prev</a> </td>
21: <th width="60%" align="center">Part I. The pkgsrc user's guide</th>
22: <td width="20%" align="right"> <a accesskey="n" href="files.html">Next</a>
23: </td>
24: </tr>
25: </table>
26: <hr>
27: </div>
28: <div class="chapter" lang="en">
29: <div class="titlepage"><div><div><h2 class="title">
30: <a name="bulk"></a>Chapter 7. Creating binary packages for everything in pkgsrc (bulk
31: builds)</h2></div></div></div>
32: <div class="toc">
33: <p><b>Table of Contents</b></p>
34: <dl>
35: <dt><span class="sect1"><a href="bulk.html#bulk.pre">7.1. Think first, build later</a></span></dt>
36: <dt><span class="sect1"><a href="bulk.html#bulk.req">7.2. Requirements of a bulk build</a></span></dt>
37: <dt><span class="sect1"><a href="bulk.html#bulk.old">7.3. Running an old-style bulk build</a></span></dt>
38: <dd><dl>
39: <dt><span class="sect2"><a href="bulk.html#binary.configuration">7.3.1. Configuration</a></span></dt>
40: <dt><span class="sect2"><a href="bulk.html#other-environmental-considerations">7.3.2. Other environmental considerations</a></span></dt>
41: <dt><span class="sect2"><a href="bulk.html#operation">7.3.3. Operation</a></span></dt>
42: <dt><span class="sect2"><a href="bulk.html#what-it-does">7.3.4. What it does</a></span></dt>
43: <dt><span class="sect2"><a href="bulk.html#disk-space-requirements">7.3.5. Disk space requirements</a></span></dt>
44: <dt><span class="sect2"><a href="bulk.html#setting-up-a-sandbox">7.3.6. Setting up a sandbox for chrooted builds</a></span></dt>
45: <dt><span class="sect2"><a href="bulk.html#building-a-partial-set">7.3.7. Building a partial set of packages</a></span></dt>
46: <dt><span class="sect2"><a href="bulk.html#bulk-upload">7.3.8. Uploading results of a bulk build</a></span></dt>
47: </dl></dd>
48: <dt><span class="sect1"><a href="bulk.html#bulk.pbulk">7.4. Running a pbulk-style bulk build</a></span></dt>
1.6 rillig 49: <dd><dl>
50: <dt><span class="sect2"><a href="bulk.html#bulk.pbulk.prepare">7.4.1. Preparation</a></span></dt>
51: <dt><span class="sect2"><a href="bulk.html#bulk.pbulk.conf">7.4.2. Configuration</a></span></dt>
52: </dl></dd>
1.1 rillig 53: <dt><span class="sect1"><a href="bulk.html#creating-cdroms">7.5. Creating a multiple CD-ROM packages collection</a></span></dt>
54: <dd><dl><dt><span class="sect2"><a href="bulk.html#cdpack-example">7.5.1. Example of cdpack</a></span></dt></dl></dd>
55: </dl>
56: </div>
57: <p>When you have multiple machines that should run the same packages,
58: it is wasted time if they all build their packages themselves from
59: source. There are two ways of getting a set of binary packages: The old
60: bulk build system, or the new (as of 2007) parallel bulk build (pbulk)
61: system. This chapter describes how to set them up so that the packages
62: are most likely to be usable later.</p>
63: <div class="sect1" lang="en">
64: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
65: <a name="bulk.pre"></a>7.1. Think first, build later</h2></div></div></div>
66: <p>Since a bulk build takes several days or even weeks to finish, you
67: should think about the setup before you start everything. Pay attention
68: to at least the following points:</p>
69: <div class="itemizedlist"><ul type="disc">
70: <li>
71: <p>If you want to upload the binary packages to
72: ftp.NetBSD.org, make sure the setup complies to the requirements for binary
73: packages:</p>
74: <div class="itemizedlist"><ul type="circle">
75: <li><p>To end up on ftp.NetBSD.org, the packages must be built
76: by a NetBSD developer on a trusted machine (that is, where you and only
77: you have root access).</p></li>
78: <li><p>Packages on ftp.NetBSD.org should only be created from
1.7 ! snj 79: the stable branches (like 2009Q1), so that users browsing the available
1.1 rillig 80: collections can see at a glance how old the packages
81: are.</p></li>
82: <li><p>The packages must be built as root, since some packages
83: require set-uid binaries at runtime, and creating those packages as
84: unprivileged user doesn't work well at the moment.</p></li>
85: </ul></div>
86: </li>
87: <li><p>Make sure that the bulk build cannot break anything in
88: your system. Most bulk builds run as root, so they should be run at least
89: in a chroot environment or something even more restrictive, depending on
90: what the operating system provides. There have been numerous cases where
91: certain packages tried to install files outside the
92: <code class="filename">LOCALBASE</code> or wanted to edit some files in
93: <code class="filename">/etc</code>. Furthermore, the bulk builds install and
94: deinstall packages in <code class="filename">/usr/pkg</code> (or whatever
95: <code class="filename">LOCALBASE</code> is) during their operation, so be sure
96: that you don't need any package during the build.</p></li>
97: </ul></div>
98: </div>
99: <div class="sect1" lang="en">
100: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
101: <a name="bulk.req"></a>7.2. Requirements of a bulk build</h2></div></div></div>
102: <p>A complete bulk build requires lots of disk space. Some of the
103: disk space can be read-only, some other must be writable. Some can be on
104: remote filesystems (such as NFS) and some should be local. Some can be
105: temporary filesystems, others must survive a sudden reboot.</p>
106: <div class="itemizedlist"><ul type="disc">
107: <li><p>10 GB for the distfiles (read-write, remote, temporary)</p></li>
108: <li><p>10 GB for the binary packages (read-write, remote, permanent)</p></li>
109: <li><p>400 MB for the pkgsrc tree (read-only, remote, permanent)</p></li>
110: <li><p>5 GB for <code class="filename">LOCALBASE</code> (read-write, local, temporary for pbulk, permanent for old-bulk)</p></li>
111: <li><p>5 GB for the log files (read-write, remote, permanent)</p></li>
112: <li><p>5 GB for temporary files (read-write, local, temporary)</p></li>
113: </ul></div>
114: </div>
115: <div class="sect1" lang="en">
116: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
117: <a name="bulk.old"></a>7.3. Running an old-style bulk build</h2></div></div></div>
1.6 rillig 118: <div class="note" style="margin-left: 0.5in; margin-right: 0.5in;">
119: <h3 class="title">Note</h3>
120: <p>There are two ways of doing a bulk build. The old-style
121: one and the new-style “<span class="quote">pbulk</span>”. The latter is the recommended
122: way.</p>
1.1 rillig 123: </div>
124: <div class="sect2" lang="en">
125: <div class="titlepage"><div><div><h3 class="title">
126: <a name="binary.configuration"></a>7.3.1. Configuration</h3></div></div></div>
127: <div class="sect3" lang="en">
128: <div class="titlepage"><div><div><h4 class="title">
129: <a name="binary.bulk.build.conf"></a>7.3.1.1. <code class="filename">build.conf</code>
130: </h4></div></div></div>
131: <p>The <code class="filename">build.conf</code> file is the main
132: configuration file for bulk builds. You can configure how your
133: copy of pkgsrc is kept up to date, how the distfiles are
134: downloaded, how the packages are built and how the report is
135: generated. You can find an annotated example file in
136: <code class="filename">pkgsrc/mk/bulk/build.conf-example</code>. To use
137: it, copy <code class="filename">build.conf-example</code> to
138: <code class="filename">build.conf</code> and edit it, following the
139: comments in that file.</p>
140: </div>
141: <div class="sect3" lang="en">
142: <div class="titlepage"><div><div><h4 class="title">
1.2 hubertf 143: <a name="binary.mk.conf"></a>7.3.1.2. <a class="link" href="configuring.html#mk.conf"><code class="filename">mk.conf</code></a>
1.1 rillig 144: </h4></div></div></div>
1.2 hubertf 145: <p>You may want to set variables in <a class="link" href="configuring.html#mk.conf"><code class="filename">mk.conf</code></a>.
1.1 rillig 146: Look at <code class="filename">pkgsrc/mk/defaults/mk.conf</code> for
147: details of the default settings. You will want to ensure that
148: <code class="varname">ACCEPTABLE_LICENSES</code> meet your local policy.
149: As used in this example, <code class="varname">_ACCEPTABLE=yes</code>
150: completely bypasses the license check.</p>
151: <pre class="programlisting">
152: PACKAGES?= ${_PKGSRCDIR}/packages/${MACHINE_ARCH}
153: WRKOBJDIR?= /usr/tmp/pkgsrc # build here instead of in pkgsrc
154: BSDSRCDIR= /usr/src
155: BSDXSRCDIR= /usr/xsrc # for x11/xservers
156: OBJHOSTNAME?= yes # use work.`hostname`
157: FAILOVER_FETCH= yes # insist on the correct checksum
158: PKG_DEVELOPER?= yes
159: _ACCEPTABLE= yes
160: </pre>
161: <p>Some options that are especially useful for bulk builds
162: can be found at the top lines of the file
163: <code class="filename">mk/bulk/bsd.bulk-pkg.mk</code>. The most useful
164: options of these are briefly described here.</p>
165: <div class="itemizedlist"><ul type="disc">
166: <li><p>If you are on a slow machine, you may want to
167: set <code class="varname">USE_BULK_BROKEN_CHECK</code> to
168: “<span class="quote">no</span>”.</p></li>
169: <li><p>If you are doing bulk builds from a read-only
170: copy of pkgsrc, you have to set <code class="varname">BULKFILESDIR</code>
171: to the directory where all log files are created. Otherwise the
172: log files are created in the pkgsrc directory.</p></li>
173: <li><p>Another important variable is
174: <code class="varname">BULK_PREREQ</code>, which is a list of packages that
175: should be always available while building other
176: packages.</p></li>
177: </ul></div>
178: <p>Some other options are scattered in the pkgsrc
179: infrastructure:</p>
180: <div class="itemizedlist"><ul type="disc">
181: <li><p><code class="varname">ALLOW_VULNERABLE_PACKAGES</code>
182: should be set to <code class="literal">yes</code>. The purpose of the bulk
183: builds is creating binary packages, no matter if they are
184: vulnerable or not. When uploading the packages to a public
185: server, the vulnerable packages will be put into a directory of
186: their own. Leaving this variable unset would prevent the bulk
187: build system from even trying to build them, so possible
188: building errors would not show up.</p></li>
189: <li><p><code class="varname">CHECK_FILES</code>
190: (<code class="filename">pkgsrc/mk/check/check-files.mk</code>) can be set to
191: “<span class="quote">yes</span>” to check that the installed set of files
192: matches the <code class="filename">PLIST</code>.</p></li>
193: <li><p><code class="varname">CHECK_INTERPRETER</code>
194: (<code class="filename">pkgsrc/mk/check/check-interpreter.mk</code>) can be set to
195: “<span class="quote">yes</span>” to check that the installed
196: “<span class="quote">#!</span>”-scripts will find their
197: interpreter.</p></li>
198: <li><p><code class="varname">PKGSRC_RUN_TEST</code> can be
199: set to “<span class="quote"><code class="literal">yes</code></span>” to run each
200: package's self-test before installing it. Note that some
201: packages make heavy use of “<span class="quote">good</span>” random
202: numbers, so you need to assure that the machine on which you
203: are doing the bulk builds is not completely idle. Otherwise
204: some test programs will seem to hang, while they are just
205: waiting for new random data to be
206: available.</p></li>
207: </ul></div>
208: </div>
209: <div class="sect3" lang="en">
210: <div class="titlepage"><div><div><h4 class="title">
211: <a name="pre-build.local"></a>7.3.1.3. <code class="filename">pre-build.local</code>
212: </h4></div></div></div>
213: <p>It is possible to configure the bulk build to perform
214: certain site-specific tasks at the end of the pre-build
215: stage. If the file
216: <code class="filename">pre-build.local</code> exists in
217: <code class="filename">/usr/pkgsrc/mk/bulk</code>, it will be executed
1.2 hubertf 218: (as a <a class="citerefentry" href="http://netbsd.gw.com/cgi-bin/man-cgi?sh+1+NetBSD-current"><span class="citerefentry"><span class="refentrytitle">sh</span>(1)</span></a> script) at the end of the usual pre-build
1.1 rillig 219: stage. An example use of
220: <code class="filename">pre-build.local</code> is to have the line:</p>
221: <pre class="screen">echo "I do not have enough disk space to build this pig." \
222: > misc/openoffice/$BROKENF</pre>
223: <p>to prevent the system from trying to build a particular package
224: which requires nearly 3 GB of disk space.</p>
225: </div>
226: </div>
227: <div class="sect2" lang="en">
228: <div class="titlepage"><div><div><h3 class="title">
229: <a name="other-environmental-considerations"></a>7.3.2. Other environmental considerations</h3></div></div></div>
230: <p>As <code class="filename">/usr/pkg</code> will be completely
231: deleted at the start of bulk builds, make sure your login
232: shell is placed somewhere else. Either drop it into
233: <code class="filename">/usr/local/bin</code> (and adjust your login
234: shell in the passwd file), or (re-)install it via
1.2 hubertf 235: <a class="citerefentry" href="http://netbsd.gw.com/cgi-bin/man-cgi?pkg_add+1+NetBSD-current"><span class="citerefentry"><span class="refentrytitle">pkg_add</span>(1)</span></a> from <code class="filename">/etc/rc.local</code>, so
1.1 rillig 236: you can login after a reboot (remember that your current
237: process won't die if the package is removed, you just can't
238: start any new instances of the shell any more). Also, if you
239: use NetBSD earlier than 1.5, or you still want to use the pkgsrc
240: version of ssh for some reason, be sure to install ssh before
241: starting it from <code class="filename">rc.local</code>:</p>
242: <pre class="programlisting">
243: (cd /usr/pkgsrc/security/ssh && make bulk-install)
244: if [ -f /usr/pkg/etc/rc.d/sshd ]; then
245: /usr/pkg/etc/rc.d/sshd
246: fi
247: </pre>
248: <p>Not doing so will result in you being not able to log in
249: via ssh after the bulk build is finished or if the machine
250: gets rebooted or crashes. You have been warned! :)</p>
251: </div>
252: <div class="sect2" lang="en">
253: <div class="titlepage"><div><div><h3 class="title">
254: <a name="operation"></a>7.3.3. Operation</h3></div></div></div>
255: <p>Make sure you don't need any of the packages still
256: installed.</p>
257: <div class="warning" style="margin-left: 0.5in; margin-right: 0.5in;">
258: <h3 class="title">Warning</h3>
259: <p>During the bulk build, <span class="emphasis"><em>all packages, their
260: configuration files and some more files from
261: <code class="filename">/var</code>, <code class="filename">/home</code> and
262: possibly other locations will be removed! So don't run a bulk
263: build with privileges that might harm your
264: system.</em></span></p>
265: </div>
266: <p>Be sure to remove all other things that might
267: interfere with builds, like some libs installed in
268: <code class="filename">/usr/local</code>, etc. then become root and type:</p>
269: <pre class="screen">
270: <code class="prompt">#</code> <strong class="userinput"><code>cd /usr/pkgsrc</code></strong>
271: <code class="prompt">#</code> <strong class="userinput"><code>sh mk/bulk/build</code></strong>
272: </pre>
273: <p>If for some reason your last build didn't complete (power
274: failure, system panic, ...), you can continue it by
275: running:</p>
276: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>sh mk/bulk/build restart</code></strong></pre>
277: <p>At the end of the bulk build, you will get a summary via mail,
278: and find build logs in the directory specified by
279: <code class="varname">FTP</code> in the <code class="filename">build.conf</code>
280: file.</p>
281: </div>
282: <div class="sect2" lang="en">
283: <div class="titlepage"><div><div><h3 class="title">
284: <a name="what-it-does"></a>7.3.4. What it does</h3></div></div></div>
285: <p>The bulk builds consist of three steps:</p>
286: <div class="variablelist"><dl>
287: <dt><span class="term">1. pre-build</span></dt>
288: <dd><p>The script updates your pkgsrc tree via (anon)cvs, then
289: cleans out any broken distfiles, and removes all
290: packages installed.</p></dd>
291: <dt><span class="term">2. the bulk build</span></dt>
292: <dd><p>This is basically “<span class="quote">make bulk-package</span>” with
293: an optimised order in which packages will be
294: built. Packages that don't require other packages will
295: be built first, and packages with many dependencies will
296: be built later.</p></dd>
297: <dt><span class="term">3. post-build</span></dt>
298: <dd><p>Generates a report that's placed in the directory
299: specified in the <code class="filename">build.conf</code> file
300: named <code class="filename">broken.html</code>, a short version
301: of that report will also be mailed to the build's
302: admin.</p></dd>
303: </dl></div>
304: <p>During the build, a list of broken packages will be compiled
305: in <code class="filename">/usr/pkgsrc/.broken</code> (or
306: <code class="filename">.../.broken.${MACHINE}</code> if
307: <code class="varname">OBJMACHINE</code> is set), individual build logs
308: of broken builds can be found in the package's
309: directory. These files are used by the bulk-targets to mark
310: broken builds to not waste time trying to rebuild them, and
311: they can be used to debug these broken package builds
312: later.</p>
313: </div>
314: <div class="sect2" lang="en">
315: <div class="titlepage"><div><div><h3 class="title">
316: <a name="disk-space-requirements"></a>7.3.5. Disk space requirements</h3></div></div></div>
317: <p>Currently, roughly the following requirements are valid for
318: NetBSD 2.0/i386:</p>
319: <div class="itemizedlist"><ul type="disc">
320: <li><p>10 GB - distfiles (NFS ok)</p></li>
321: <li><p>8 GB - full set of all binaries (NFS ok)</p></li>
322: <li><p>5 GB - temp space for compiling (local disk recommended)</p></li>
323: </ul></div>
324: <p>Note that all pkgs will be de-installed as soon as they are
325: turned into a binary package, and that sources are removed,
326: so there is no excessively huge demand to disk
327: space. Afterwards, if the package is needed again, it will
1.2 hubertf 328: be installed via <a class="citerefentry" href="http://netbsd.gw.com/cgi-bin/man-cgi?pkg_add+1+NetBSD-current"><span class="citerefentry"><span class="refentrytitle">pkg_add</span>(1)</span></a> instead of building again, so
1.1 rillig 329: there are no cycles wasted by recompiling.</p>
330: </div>
331: <div class="sect2" lang="en">
332: <div class="titlepage"><div><div><h3 class="title">
333: <a name="setting-up-a-sandbox"></a>7.3.6. Setting up a sandbox for chrooted builds</h3></div></div></div>
334: <p>If you don't want all the packages nuked from a machine
335: (rendering it useless for anything but pkg compiling), there
336: is the possibility of doing the package bulk build inside a
337: chroot environment.</p>
338: <p>The first step is to set up a chroot sandbox,
339: e.g. <code class="filename">/usr/sandbox</code>. This can be done by
340: using null mounts, or manually.</p>
341: <p>There is a shell script called
342: <code class="filename">pkgsrc/mk/bulk/mksandbox</code> which will set
343: up the sandbox environment using null mounts. It will also
344: create a script called <code class="filename">sandbox</code> in the
345: root of the sandbox environment, which will allow the null
1.2 hubertf 346: mounts to be activated using the <span class="command"><strong>sandbox
1.1 rillig 347: mount</strong></span> command and deactivated using the
1.2 hubertf 348: <span class="command"><strong>sandbox umount</strong></span> command.</p>
1.1 rillig 349: <p>To set up a sandbox environment by hand, after extracting all
1.2 hubertf 350: the sets from a NetBSD installation or doing a <span class="command"><strong>make
1.1 rillig 351: distribution DESTDIR=/usr/sandbox</strong></span> in
352: <code class="filename">/usr/src/etc</code>, be sure the following items
353: are present and properly configured:</p>
354: <div class="procedure"><ol type="1">
355: <li>
356: <p>Kernel</p>
357: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>cp /netbsd /usr/sandbox</code></strong></pre>
358: </li>
359: <li>
360: <p><code class="filename">/dev/*</code></p>
361: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>cd /usr/sandbox/dev ; sh MAKEDEV all</code></strong></pre>
362: </li>
363: <li>
364: <p><code class="filename">/etc/resolv.conf</code> (for <a href="ftp://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc/security/smtpd/README.html" target="_top"><code class="filename">security/smtpd</code></a> and mail):</p>
365: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>cp /etc/resolv.conf /usr/sandbox/etc</code></strong></pre>
366: </li>
367: <li>
368: <p>Working(!) mail config (hostname, sendmail.cf):</p>
369: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>cp /etc/mail/sendmail.cf /usr/sandbox/etc/mail</code></strong></pre>
370: </li>
371: <li>
372: <p><code class="filename">/etc/localtime</code> (for <a href="ftp://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc/security/smtpd/README.html" target="_top"><code class="filename">security/smtpd</code></a>):</p>
373: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>ln -sf /usr/share/zoneinfo/UTC /usr/sandbox/etc/localtime</code></strong></pre>
374: </li>
375: <li>
376: <p><code class="filename">/usr/src</code> (system sources,
1.5 jmcneill 377: e. g. for <a href="ftp://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc/sysutils/aperture/README.html" target="_top"><code class="filename">sysutils/aperture</code></a>):</p>
1.1 rillig 378: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>ln -s ../disk1/cvs .</code></strong>
379: <code class="prompt">#</code> <strong class="userinput"><code>ln -s cvs/src-2.0 src</code></strong></pre>
380: </li>
381: <li>
382: <p>Create <code class="filename">/var/db/pkg</code> (not part of default install):</p>
383: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>mkdir /usr/sandbox/var/db/pkg</code></strong></pre>
384: </li>
385: <li>
386: <p>Create <code class="filename">/usr/pkg</code> (not part of default install):</p>
387: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>mkdir /usr/sandbox/usr/pkg</code></strong></pre>
388: </li>
389: <li>
390: <p>Checkout pkgsrc via cvs into
391: <code class="filename">/usr/sandbox/usr/pkgsrc</code>:</p>
392: <pre class="screen">
393: <code class="prompt">#</code> <strong class="userinput"><code>cd /usr/sandbox/usr</code></strong>
394: <code class="prompt">#</code> <strong class="userinput"><code>cvs -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -d -P pkgsrc</code></strong>
395: </pre>
396: <p>Do not mount/link this to the copy of your pkgsrc tree
397: you do development in, as this will likely cause problems!</p>
398: </li>
399: <li><p>Make
400: <code class="filename">/usr/sandbox/usr/pkgsrc/packages</code> and
401: <code class="filename">.../distfiles</code> point somewhere
402: appropriate. NFS- and/or nullfs-mounts may come in handy!</p></li>
1.6 rillig 403: <li><p>Edit <a class="link" href="configuring.html#mk.conf"><code class="filename">mk.conf</code></a>, see <a class="xref" href="bulk.html#binary.mk.conf" title="7.3.1.2. ???">Section 7.3.1.2, “???”</a>.</p></li>
1.1 rillig 404: <li><p>Adjust <code class="filename">mk/bulk/build.conf</code> to suit your needs.</p></li>
405: </ol></div>
406: <p>When the chroot sandbox is set up, you can start
407: the build with the following steps:</p>
408: <pre class="screen">
409: <code class="prompt">#</code> <strong class="userinput"><code>cd /usr/sandbox/usr/pkgsrc</code></strong>
410: <code class="prompt">#</code> <strong class="userinput"><code>sh mk/bulk/do-sandbox-build</code></strong>
411: </pre>
412: <p>This will just jump inside the sandbox and start building. At
413: the end of the build, mail will be sent with the results of
414: the build. Created binary pkgs will be in
415: <code class="filename">/usr/sandbox/usr/pkgsrc/packages</code>
416: (wherever that points/mounts to/from).</p>
417: </div>
418: <div class="sect2" lang="en">
419: <div class="titlepage"><div><div><h3 class="title">
420: <a name="building-a-partial-set"></a>7.3.7. Building a partial set of packages</h3></div></div></div>
421: <p>In addition to building a complete set of all packages in
422: pkgsrc, the <code class="filename">pkgsrc/mk/bulk/build</code> script
423: may be used to build a subset of the packages contained in
424: pkgsrc. By setting <code class="varname">SPECIFIC_PKGS</code>
1.2 hubertf 425: in <a class="link" href="configuring.html#mk.conf"><code class="filename">mk.conf</code></a>, the variables</p>
1.1 rillig 426: <div class="itemizedlist"><ul type="disc">
427: <li><p>SITE_SPECIFIC_PKGS</p></li>
428: <li><p>HOST_SPECIFIC_PKGS</p></li>
429: <li><p>GROUP_SPECIFIC_PKGS</p></li>
430: <li><p>USER_SPECIFIC_PKGS</p></li>
431: </ul></div>
432: <p>will define the set of packages which should be built.
433: The bulk build code will also include any packages which are
434: needed as dependencies for the explicitly listed packages.</p>
435: <p>One use of this is to do a bulk build with
436: <code class="varname">SPECIFIC_PKGS</code> in a chroot sandbox
437: periodically to have a complete set of the binary packages
438: needed for your site available without the overhead of
439: building extra packages that are not needed.</p>
440: </div>
441: <div class="sect2" lang="en">
442: <div class="titlepage"><div><div><h3 class="title">
443: <a name="bulk-upload"></a>7.3.8. Uploading results of a bulk build</h3></div></div></div>
444: <p>This section describes how pkgsrc developers can upload binary
445: pkgs built by bulk builds to ftp.NetBSD.org.</p>
446: <p>If you would like to automatically create checksum files for the
447: binary packages you intend to upload, remember to set
448: <code class="varname">MKSUMS=yes</code> in your
449: <code class="filename">mk/bulk/build.conf</code>.</p>
450: <p>If you would like to PGP sign the checksum files (highly
451: recommended!), remember to set
452: <code class="varname">SIGN_AS=username@NetBSD.org</code> in your
453: <code class="filename">mk/bulk/build.conf</code>. This will prompt you for
454: your GPG password to sign the files before uploading everything.</p>
455: <p>Then, make sure that you have <code class="varname">RSYNC_DST</code>
456: set properly in your <code class="filename">mk/bulk/build.conf</code>
457: file, i.e. adjust it to something like one of the following:</p>
458: <pre class="screen">RSYNC_DST=ftp.NetBSD.org:/pub/NetBSD/packages/packages-200xQy/NetBSD-a.b.c/arch/upload </pre>
459: <p>Please use appropriate values for "packages-200xQy",
460: "NetBSD-a.b.c" and "arch" here. If your login on ftp.NetBSD.org
461: is different from your local login, write your login directly
462: into the variable, e.g. my local account is "feyrer", but for my
463: login "hubertf", I use:</p>
464: <pre class="screen">RSYNC_DST=hubertf@ftp.NetBSD.org:/pub/NetBSD/packages/packages-200xQy/NetBSD-a.b.c/arch/upload</pre>
465: <p>A separate <code class="filename">upload</code> directory is used
466: here to allow "closing" the directory during upload. To do
467: so, run the following command on ftp.NetBSD.org next:</p>
468: <pre class="screen">nbftp% <strong class="userinput"><code>mkdir -p -m 750 /pub/NetBSD/packages/packages-200xQy/NetBSD-a.b.c/arch/upload</code></strong></pre>
469: <p>Please note that <code class="filename">/pub/NetBSD/packages</code> is
470: only appropriate for packages for the NetBSD operating
471: system. Binary packages for other operating systems should go
472: into <code class="filename">/pub/pkgsrc</code>.</p>
473: <p>Before uploading the binary pkgs, ssh authentication needs
474: to be set up. This example shows how to set up temporary keys
475: for the root account <span class="emphasis"><em>inside the sandbox</em></span>
476: (assuming that no keys should be present there usually):</p>
477: <pre class="screen">
478: <code class="prompt">#</code> <strong class="userinput"><code>chroot /usr/sandbox</code></strong>
479: chroot-<code class="prompt">#</code> <strong class="userinput"><code>rm $HOME/.ssh/id-dsa*</code></strong>
480: chroot-<code class="prompt">#</code> <strong class="userinput"><code>ssh-keygen -t dsa</code></strong>
481: chroot-<code class="prompt">#</code> <strong class="userinput"><code>cat $HOME/.ssh/id-dsa.pub</code></strong>
482: </pre>
483: <p>Now take the output of <code class="filename">id-dsa.pub</code> and
484: append it to your <code class="filename">~/.ssh/authorized_keys</code>
485: file on ftp.NetBSD.org. You can remove the key after the
486: upload is done!</p>
487: <p>Next, test if your ssh connection really works:</p>
488: <pre class="screen">chroot-<code class="prompt">#</code> <strong class="userinput"><code>ssh ftp.NetBSD.org date</code></strong> </pre>
489: <p>Use "-l yourNetBSDlogin" here as appropriate!</p>
490: <p>Now after all this works, you can exit the sandbox and start
491: the upload:</p>
492: <pre class="screen">
493: chroot-<code class="prompt">#</code> <strong class="userinput"><code>exit</code></strong>
494: <code class="prompt">#</code> <strong class="userinput"><code>cd /usr/sandbox/usr/pkgsrc</code></strong>
495: <code class="prompt">#</code> <strong class="userinput"><code>sh mk/bulk/do-sandbox-upload</code></strong>
496: </pre>
1.2 hubertf 497: <p>The upload process may take quite some time. Use <a class="citerefentry" href="http://netbsd.gw.com/cgi-bin/man-cgi?ls+1+NetBSD-current"><span class="citerefentry"><span class="refentrytitle">ls</span>(1)</span></a> or
498: <a class="citerefentry" href="http://netbsd.gw.com/cgi-bin/man-cgi?du+1+NetBSD-current"><span class="citerefentry"><span class="refentrytitle">du</span>(1)</span></a> on the FTP server to monitor progress of the
1.1 rillig 499: upload. The upload script will take care of not uploading
500: restricted packages and putting vulnerable packages into the
501: <code class="filename">vulnerable</code> subdirectory.</p>
502: <p>After the upload has ended, first thing is to revoke ssh access:</p>
503: <pre class="screen">nbftp% <strong class="userinput"><code>vi ~/.ssh/authorized_keys</code></strong>
504: Gdd:x! </pre>
505: <p>Use whatever is needed to remove the key you've entered
506: before! Last, move the uploaded packages out of the
507: <code class="filename">upload</code> directory to have them accessible
508: to everyone:</p>
509: <pre class="screen">
510: nbftp% <strong class="userinput"><code>cd /pub/NetBSD/packages/packages-200xQy/NetBSD-a.b.c/arch</code></strong>
511: nbftp% <strong class="userinput"><code>mv upload/* .</code></strong>
512: nbftp% <strong class="userinput"><code>rmdir upload</code></strong>
513: nbftp% <strong class="userinput"><code>chmod 755 .</code></strong>
514: </pre>
515: </div>
516: </div>
517: <div class="sect1" lang="en">
518: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
519: <a name="bulk.pbulk"></a>7.4. Running a pbulk-style bulk build</h2></div></div></div>
1.6 rillig 520: <p>Running a pbulk-style bulk build works roughly as follows:</p>
521: <div class="itemizedlist"><ul type="disc">
522: <li><p>First, build the pbulk infrastructure in a fresh pkgsrc location.</p></li>
523: <li><p>Then, build each of the packages from a clean installation directory using the infrastructure.</p></li>
524: </ul></div>
525: <div class="sect2" lang="en">
526: <div class="titlepage"><div><div><h3 class="title">
527: <a name="bulk.pbulk.prepare"></a>7.4.1. Preparation</h3></div></div></div>
528: <p>First, you need to create a pkgsrc installation for the pbulk infrastructure. No matter on which platform you are (even on NetBSD), you should bootstrap into its own directory. Let's take the directory <code class="filename">/usr/pbulk</code> or <code class="filename">$HOME/pbulk</code> for it. This installation will be bootstrapped and all the tools that are required for the bulk build will be installed there.</p>
529: <pre class="screen">
530: $ <strong class="userinput"><code>cd /usr/pkgsrc</code></strong>
531: $ <strong class="userinput"><code>./bootstrap/bootstrap --prefix=/usr/pbulk --varbase=/usr/pbulk/var --workdir=/tmp/pbulk-bootstrap</code></strong>
532: $ <strong class="userinput"><code>rm -rf /tmp/pbulk-bootstrap</code></strong>
533: </pre>
534: <p>Now the basic environment for the pbulk infrastructure is installed. The specific tools are still missing. This is a good time to edit the pkgsrc configuration file <code class="filename">/usr/pbulk/etc/mk.conf</code> to fit your needs. Typical things you might set now are:</p>
535: <div class="itemizedlist"><ul type="disc">
536: <li><p><code class="literal"><code class="varname">PKG_DEVELOPER</code>=yes</code>, to enable many consistency checks,</p></li>
537: <li><p><code class="literal"><code class="varname">WRKOBJDIR</code>=/tmp/pbulk-outer</code>, to keep <code class="filename">/usr/pkgsrc</code> free from any modifications,</p></li>
538: <li><p><code class="literal"><code class="varname">DISTDIR</code>=/distfiles</code>, to have only one directory in which all distfiles (for the infrastructure and for the actual packages) are downloaded,</p></li>
539: <li><p><code class="literal"><code class="varname">ACCEPTABLE_LICENSES</code>+=...</code>, to select some licenses additional to the usual Free/Open Source licenses that are acceptable to you,</p></li>
540: <li><p><code class="literal"><code class="varname">_ACCEPTABLE</code>=yes</code>, to accept all licenses, no matter how restrictive they are.</p></li>
541: </ul></div>
542: <p>Now you are ready to build the rest of the pbulk infrastructure.</p>
543: <pre class="screen">
544: $ <strong class="userinput"><code>cd pkgtools/pbulk</code></strong>
545: $ <strong class="userinput"><code>/usr/pbulk/bin/bmake install</code></strong>
546: $ <strong class="userinput"><code>rm -rf /tmp/pbulk-outer</code></strong>
547: </pre>
548: <p>Now the pbulk infrastructure is built and installed. It still needs to be configured, and after some more preparation, we will be able to start the real bulk build.</p>
549: </div>
1.1 rillig 550: <div class="sect2" lang="en">
551: <div class="titlepage"><div><div><h3 class="title">
1.6 rillig 552: <a name="bulk.pbulk.conf"></a>7.4.2. Configuration</h3></div></div></div>
1.5 jmcneill 553: <p>TODO; see pkgsrc/doc/HOWTO-pbulk for more information.</p>
1.6 rillig 554: <p>TODO: continue writing</p>
1.1 rillig 555: </div>
556: </div>
557: <div class="sect1" lang="en">
558: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
559: <a name="creating-cdroms"></a>7.5. Creating a multiple CD-ROM packages collection</h2></div></div></div>
560: <p>After your pkgsrc bulk-build has completed, you may wish to
561: create a CD-ROM set of the resulting binary packages to assist
562: in installing packages on other machines. The
563: <a href="ftp://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc/pkgtools/cdpack/README.html" target="_top"><code class="filename">pkgtools/cdpack</code></a> package provides
564: a simple tool for creating the ISO 9660 images.
1.2 hubertf 565: <span class="command"><strong>cdpack</strong></span> arranges the packages on the CD-ROMs in a
1.1 rillig 566: way that keeps all the dependencies for a given package on the same
567: CD as that package.</p>
568: <div class="sect2" lang="en">
569: <div class="titlepage"><div><div><h3 class="title">
570: <a name="cdpack-example"></a>7.5.1. Example of cdpack</h3></div></div></div>
571: <p>Complete documentation for cdpack is found in the cdpack(1)
572: man page. The following short example assumes that the binary
573: packages are left in
574: <code class="filename">/usr/pkgsrc/packages/All</code> and that
575: sufficient disk space exists in <code class="filename">/u2</code> to
576: hold the ISO 9660 images.</p>
577: <pre class="screen">
578: <code class="prompt">#</code> <strong class="userinput"><code>mkdir /u2/images</code></strong>
579: <code class="prompt">#</code> <strong class="userinput"><code>pkg_add /usr/pkgsrc/packages/All/cdpack</code></strong>
580: <code class="prompt">#</code> <strong class="userinput"><code>cdpack /usr/pkgsrc/packages/All /u2/images</code></strong>
581: </pre>
582: <p>If you wish to include a common set of files
583: (<code class="filename">COPYRIGHT</code>, <code class="filename">README</code>,
584: etc.) on each CD in the collection, then you need to create a
585: directory which contains these files. e.g.</p>
586: <pre class="screen">
587: <code class="prompt">#</code> <strong class="userinput"><code>mkdir /tmp/common</code></strong>
588: <code class="prompt">#</code> <strong class="userinput"><code>echo "This is a README" > /tmp/common/README</code></strong>
589: <code class="prompt">#</code> <strong class="userinput"><code>echo "Another file" > /tmp/common/COPYING</code></strong>
590: <code class="prompt">#</code> <strong class="userinput"><code>mkdir /tmp/common/bin</code></strong>
591: <code class="prompt">#</code> <strong class="userinput"><code>echo "#!/bin/sh" > /tmp/common/bin/myscript</code></strong>
592: <code class="prompt">#</code> <strong class="userinput"><code>echo "echo Hello world" >> /tmp/common/bin/myscript</code></strong>
593: <code class="prompt">#</code> <strong class="userinput"><code>chmod 755 /tmp/common/bin/myscript</code></strong>
594: </pre>
595: <p>Now create the images:</p>
596: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images</code></strong></pre>
597: <p>Each image will contain <code class="filename">README</code>,
598: <code class="filename">COPYING</code>, and <code class="filename">bin/myscript</code>
599: in their root directories.</p>
600: </div>
601: </div>
602: </div>
603: <div class="navfooter">
604: <hr>
605: <table width="100%" summary="Navigation footer">
606: <tr>
607: <td width="40%" align="left">
608: <a accesskey="p" href="binary.html">Prev</a> </td>
609: <td width="20%" align="center"><a accesskey="u" href="users-guide.html">Up</a></td>
610: <td width="40%" align="right"> <a accesskey="n" href="files.html">Next</a>
611: </td>
612: </tr>
613: <tr>
614: <td width="40%" align="left" valign="top">Chapter 6. Creating binary packages </td>
615: <td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td>
616: <td width="40%" align="right" valign="top"> Chapter 8. Directory layout of the installed files</td>
617: </tr>
618: </table>
619: </div>
620: </body>
621: </html>
CVSweb <webmaster@jp.NetBSD.org>