Annotation of htdocs/docs/pkgsrc/bulk.html, revision 1.26
1.1 rillig 1: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
2: <html>
3: <head>
4: <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
5: <title>Chapter 7. Creating binary packages for everything in pkgsrc (bulk builds)</title>
1.10 spz 6: <link rel="stylesheet" type="text/css" href="/global.css">
1.1 rillig 7: <meta name="generator" content="DocBook XSL Stylesheets VX.X.X">
1.6 rillig 8: <link rel="home" href="index.html" title="The pkgsrc guide">
1.1 rillig 9: <link rel="up" href="users-guide.html" title="Part I. The pkgsrc user's guide">
10: <link rel="prev" href="binary.html" title="Chapter 6. Creating binary packages">
11: <link rel="next" href="files.html" title="Chapter 8. Directory layout of the installed files">
12: </head>
13: <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF">
14: <div class="navheader">
15: <table width="100%" summary="Navigation header">
16: <tr><th colspan="3" align="center">Chapter 7. Creating binary packages for everything in pkgsrc (bulk
17: builds)</th></tr>
18: <tr>
19: <td width="20%" align="left">
20: <a accesskey="p" href="binary.html">Prev</a> </td>
21: <th width="60%" align="center">Part I. The pkgsrc user's guide</th>
22: <td width="20%" align="right"> <a accesskey="n" href="files.html">Next</a>
23: </td>
24: </tr>
25: </table>
26: <hr>
27: </div>
1.13 asau 28: <div class="chapter">
1.1 rillig 29: <div class="titlepage"><div><div><h2 class="title">
30: <a name="bulk"></a>Chapter 7. Creating binary packages for everything in pkgsrc (bulk
31: builds)</h2></div></div></div>
32: <div class="toc">
33: <p><b>Table of Contents</b></p>
1.26 ! sevan 34: <dl>
1.19 wiz 35: <dt><span class="sect1"><a href="bulk.html#bulk.pre">7.1. Preparations</a></span></dt>
36: <dt><span class="sect1"><a href="bulk.html#bulk.pbulk">7.2. Running a pbulk-style bulk build</a></span></dt>
37: <dd><dl><dt><span class="sect2"><a href="bulk.html#bulk.pbulk.conf">7.2.1. Configuration</a></span></dt></dl></dd>
38: <dt><span class="sect1"><a href="bulk.html#bulk.req">7.3. Requirements of a full bulk build</a></span></dt>
1.16 asau 39: <dt><span class="sect1"><a href="bulk.html#creating-cdroms">7.4. Creating a multiple CD-ROM packages collection</a></span></dt>
40: <dd><dl><dt><span class="sect2"><a href="bulk.html#cdpack-example">7.4.1. Example of cdpack</a></span></dt></dl></dd>
1.1 rillig 41: </dl>
42: </div>
1.19 wiz 43: <p>For a number of reasons you may want to build binary packages
44: for a large selected set of packages in pkgsrc or even for all pkgsrc packages.
45: For instance, when you have multiple machines that should run the same software,
46: it is wasted time if they all build their packages themselves from source.
47: Or you may want to build a list of packages you want and check them before
48: deploying onto production system.
49: There is a way of getting a set of binary packages:
1.16 asau 50: The bulk build system, or pbulk ("p" stands for "parallel).
51: This chapter describes how to set it up so that the packages
1.1 rillig 52: are most likely to be usable later.</p>
1.13 asau 53: <div class="sect1">
1.1 rillig 54: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
1.19 wiz 55: <a name="bulk.pre"></a>7.1. Preparations</h2></div></div></div>
56: <p>First of all, you have to decide whether you build all packages
57: or a limited set of them. Full bulk builds usually consume a lot more resources,
58: both space and time, than builds for some practical sets of packages.
59: There exists a number of particularly heavy packages that are not actually
60: interesting to a wide audience.
61:
62: For a limited bulk builds you need to make a list of packages you want to build.
63: Note, that all their dependencies will be built, so you don't need to track them manually.
64: </p>
65: <p>During bulk builds various packages are installed and deinstalled
66: in <code class="filename">/usr/pkg</code> (or whatever <code class="filename">LOCALBASE</code> is),
67: so make sure that you don't need any package during the builds.
68: Essentially, you should provide fresh system, either a chroot environment
69: or something even more restrictive, depending on what the operating system provides,
70: or dedicate the whole physical machine.
71: As a useful side effect this makes sure that bulk builds cannot
72: break anything in your system. There have been numerous cases where
1.1 rillig 73: certain packages tried to install files outside the
74: <code class="filename">LOCALBASE</code> or wanted to edit some files in
1.19 wiz 75: <code class="filename">/etc</code>.
76: </p>
1.1 rillig 77: </div>
1.13 asau 78: <div class="sect1">
1.1 rillig 79: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
1.19 wiz 80: <a name="bulk.pbulk"></a>7.2. Running a pbulk-style bulk build</h2></div></div></div>
1.6 rillig 81: <p>Running a pbulk-style bulk build works roughly as follows:</p>
1.13 asau 82: <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
1.9 wiz 83: <li class="listitem"><p>First, build the pbulk infrastructure in a fresh pkgsrc location.</p></li>
84: <li class="listitem"><p>Then, build each of the packages from a clean installation directory using the infrastructure.</p></li>
1.6 rillig 85: </ul></div>
1.13 asau 86: <div class="sect2">
1.6 rillig 87: <div class="titlepage"><div><div><h3 class="title">
1.19 wiz 88: <a name="bulk.pbulk.conf"></a>7.2.1. Configuration</h3></div></div></div>
1.17 asau 89: <p>To simplify configuration we provide helper script <code class="filename">mk/pbulk/pbulk.sh</code>.</p>
90: <p>In order to use it, prepare a clear system (real one, chroot environment, jail, zone, virtual machine).
91: Configure network access to fetch distribution files.
92: Create user with name "pbulk".</p>
93: <p>Fetch and extract pkgsrc. Use a command like one of these:</p>
94: <pre class="screen">
1.20 wiz 95: <code class="prompt">#</code> <strong class="userinput"><code>(cd /usr && ftp -o - http://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)</code></strong>
96: <code class="prompt">#</code> <strong class="userinput"><code>(cd /usr && fetch -o - http://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc.tar.gz | tar -zxf-)</code></strong>
97: <code class="prompt">#</code> <strong class="userinput"><code>(cd /usr && cvs -Q -z3 -d anoncvs@anoncvs.NetBSD.org:/cvsroot get -P pkgsrc)</code></strong>
1.17 asau 98: </pre>
99: <p>Or any other way that fits (e.g., curl, wget).</p>
100: <p>Deploy and configure pbulk tools, e.g.:</p>
101: <pre class="screen">
102: <code class="prompt">#</code> <strong class="userinput"><code>sh pbulk.sh -n # native (NetBSD)</code></strong>
103: <code class="prompt">#</code> <strong class="userinput"><code>sh pbulk.sh -n -c mk.conf.frag # native, apply settings from given mk.conf fragment</code></strong>
104: <code class="prompt">#</code> <strong class="userinput"><code>sh pbulk.sh -nlc mk.conf.frag # native, apply settings, configure for limited build</code></strong>
105: </pre>
106: <div class="note" style="margin-left: 0.5in; margin-right: 0.5in;">
107: <h3 class="title">Note</h3>
108: <p><code class="filename">mk.conf.frag</code> is a fragment of
109: <code class="filename">mk.conf</code> that contains settings you want to
110: apply to packages you build. For instance,</p>
111: <pre class="programlisting">
112: PKG_DEVELOPER= yes # perform more checks
113: X11_TYPE= modular # use pkgsrc X11
114: SKIP_LICENSE_CHECK= yes # accept all licences (useful when building all packages)
115: </pre>
116: </div>
117: <p>If configured for limited list, replace the list in <code class="filename">/usr/pbulk/etc/pbulk.list</code>
118: with your list of packages one per line without empty lines or comments. E.g.:</p>
119: <pre class="programlisting">
120: www/firefox
121: mail/thunderbird
122: misc/libreoffice4
123: </pre>
1.18 asau 124: <p>At this point you can also review configuration in <code class="filename">/usr/pbulk/etc</code>
125: and make final amendments, if wanted.</p>
1.17 asau 126: <p>Start it:</p>
127: <pre class="screen">
128: <code class="prompt">#</code> <strong class="userinput"><code>/usr/pbulk/bin/bulkbuild</code></strong>
129: </pre>
130: <p>After it finishes, you'll have <code class="filename">/mnt</code> filled with distribution files, binary packages, and reports,
131: plain text summary in <code class="filename">/mnt/bulklog/meta/report.txt</code>
132: </p>
133: <div class="note" style="margin-left: 0.5in; margin-right: 0.5in;">
134: <h3 class="title">Note</h3>
135: <p>The <code class="filename">pbulk.sh</code> script does not cover all possible use cases.
136: While being ready to run, it serves as a good starting point to understand and build more complex setups.
137: The script is kept small enough for better understanding.</p>
138: </div>
139: <div class="note" style="margin-left: 0.5in; margin-right: 0.5in;">
140: <h3 class="title">Note</h3>
141: <p>The <code class="filename">pbulk.sh</code> script supports running
142: unprivileged bulk build and helps configuring distributed bulk builds.</p>
143: </div>
1.1 rillig 144: </div>
145: </div>
1.13 asau 146: <div class="sect1">
1.1 rillig 147: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
1.19 wiz 148: <a name="bulk.req"></a>7.3. Requirements of a full bulk build</h2></div></div></div>
149: <p>A complete bulk build requires lots of disk space. Some of the
150: disk space can be read-only, some other must be writable. Some can be on
151: remote filesystems (such as NFS) and some should be local. Some can be
152: temporary filesystems, others must survive a sudden reboot.</p>
153: <div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; ">
154: <li class="listitem"><p>40 GB for the distfiles (read-write, remote, temporary)</p></li>
155: <li class="listitem"><p>30 GB for the binary packages (read-write, remote, permanent)</p></li>
156: <li class="listitem"><p>1 GB for the pkgsrc tree (read-only, remote, permanent)</p></li>
157: <li class="listitem"><p>5 GB for <code class="filename">LOCALBASE</code> (read-write, local, temporary)</p></li>
158: <li class="listitem"><p>10 GB for the log files (read-write, remote, permanent)</p></li>
159: <li class="listitem"><p>5 GB for temporary files (read-write, local, temporary)</p></li>
160: </ul></div>
161: </div>
162: <div class="sect1">
163: <div class="titlepage"><div><div><h2 class="title" style="clear: both">
1.16 asau 164: <a name="creating-cdroms"></a>7.4. Creating a multiple CD-ROM packages collection</h2></div></div></div>
1.1 rillig 165: <p>After your pkgsrc bulk-build has completed, you may wish to
166: create a CD-ROM set of the resulting binary packages to assist
167: in installing packages on other machines. The
168: <a href="ftp://ftp.NetBSD.org/pub/pkgsrc/current/pkgsrc/pkgtools/cdpack/README.html" target="_top"><code class="filename">pkgtools/cdpack</code></a> package provides
169: a simple tool for creating the ISO 9660 images.
1.2 hubertf 170: <span class="command"><strong>cdpack</strong></span> arranges the packages on the CD-ROMs in a
1.1 rillig 171: way that keeps all the dependencies for a given package on the same
172: CD as that package.</p>
1.13 asau 173: <div class="sect2">
1.1 rillig 174: <div class="titlepage"><div><div><h3 class="title">
1.16 asau 175: <a name="cdpack-example"></a>7.4.1. Example of cdpack</h3></div></div></div>
1.1 rillig 176: <p>Complete documentation for cdpack is found in the cdpack(1)
177: man page. The following short example assumes that the binary
178: packages are left in
179: <code class="filename">/usr/pkgsrc/packages/All</code> and that
180: sufficient disk space exists in <code class="filename">/u2</code> to
181: hold the ISO 9660 images.</p>
182: <pre class="screen">
183: <code class="prompt">#</code> <strong class="userinput"><code>mkdir /u2/images</code></strong>
184: <code class="prompt">#</code> <strong class="userinput"><code>pkg_add /usr/pkgsrc/packages/All/cdpack</code></strong>
185: <code class="prompt">#</code> <strong class="userinput"><code>cdpack /usr/pkgsrc/packages/All /u2/images</code></strong>
186: </pre>
187: <p>If you wish to include a common set of files
188: (<code class="filename">COPYRIGHT</code>, <code class="filename">README</code>,
189: etc.) on each CD in the collection, then you need to create a
190: directory which contains these files. e.g.</p>
191: <pre class="screen">
192: <code class="prompt">#</code> <strong class="userinput"><code>mkdir /tmp/common</code></strong>
193: <code class="prompt">#</code> <strong class="userinput"><code>echo "This is a README" > /tmp/common/README</code></strong>
194: <code class="prompt">#</code> <strong class="userinput"><code>echo "Another file" > /tmp/common/COPYING</code></strong>
195: <code class="prompt">#</code> <strong class="userinput"><code>mkdir /tmp/common/bin</code></strong>
196: <code class="prompt">#</code> <strong class="userinput"><code>echo "#!/bin/sh" > /tmp/common/bin/myscript</code></strong>
197: <code class="prompt">#</code> <strong class="userinput"><code>echo "echo Hello world" >> /tmp/common/bin/myscript</code></strong>
198: <code class="prompt">#</code> <strong class="userinput"><code>chmod 755 /tmp/common/bin/myscript</code></strong>
199: </pre>
200: <p>Now create the images:</p>
201: <pre class="screen"><code class="prompt">#</code> <strong class="userinput"><code>cdpack -x /tmp/common /usr/pkgsrc/packages/All /u2/images</code></strong></pre>
202: <p>Each image will contain <code class="filename">README</code>,
203: <code class="filename">COPYING</code>, and <code class="filename">bin/myscript</code>
204: in their root directories.</p>
205: </div>
206: </div>
207: </div>
208: <div class="navfooter">
209: <hr>
210: <table width="100%" summary="Navigation footer">
211: <tr>
212: <td width="40%" align="left">
213: <a accesskey="p" href="binary.html">Prev</a> </td>
214: <td width="20%" align="center"><a accesskey="u" href="users-guide.html">Up</a></td>
215: <td width="40%" align="right"> <a accesskey="n" href="files.html">Next</a>
216: </td>
217: </tr>
218: <tr>
219: <td width="40%" align="left" valign="top">Chapter 6. Creating binary packages </td>
220: <td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td>
221: <td width="40%" align="right" valign="top"> Chapter 8. Directory layout of the installed files</td>
222: </tr>
223: </table>
224: </div>
225: </body>
226: </html>
CVSweb <webmaster@jp.NetBSD.org>