File: [cvs.NetBSD.org] / htdocs / gallery / 10years.xml (download)
Revision 1.13, Thu Aug 9 13:14:54 2012 UTC (11 years, 7 months ago) by abs
Branch: MAIN
CVS Tags: HEAD Changes since 1.12: +4 -4
lines
Drop unnecessary "." and replace double space with space - from IRC
|
<?xml version="1.0"?>
<!DOCTYPE webpage
PUBLIC "-//NetBSD//DTD Website-based NetBSD Extension//EN"
"http://www.NetBSD.org/XML/htdocs/lang/share/xml/website-netbsd.dtd">
<webpage id="gallery-10years">
<config param="desc" value="10 Years of pkgsrc!"/>
<config param="cvstag" value="$NetBSD: 10years.xml,v 1.13 2012/08/09 13:14:54 abs Exp $"/>
<config param="rcsdate" value="$Date: 2012/08/09 13:14:54 $"/>
<head>
<title>10 years of pkgsrc - pkgsrc and the concepts of package management 1997-2007 (part 1)</title>
</head>
<sect1 id="intro">
<para>
On October 3rd 1997, the pkgsrc software management system
was created by Alistair Crooks and Hubert Feyrer. The
<emphasis>NetBSD Packages Collection</emphasis>, was intended primarily
as a packaging system for NetBSD. Derived from the FreeBSD Ports
system, pkgsrc became a success story. Today, pkgsrc is a
cross-platform framework, running on the BSDs, Linux, Solaris,
Mac OS X, many Unix derivatives, and even on QNX and Windows.</para>
<para>Ports- and pkgsrc-like software build frameworks are today standard on
the BSDs and quite popular on some newer Linux distributions. In 2005
pkgsrc was adopted as the package management system for
<emphasis>DragonFly BSD</emphasis>.</para>
<para>
The pkgsrc project is based on active development
and permanent improvements. The developers join once-a-year in Europe on
the <firstterm>pkgsrcCon</firstterm> conference discussing problems and innovations,
sharing experiences and new concepts.</para>
<para>In the following interviews, developers and users of pkgsrc and of related systems
talk about the history, the concepts, the problems and the future directions of packaging systems.
This is the first part, the follow-up with more interviews is
<ulink url="pkgsrc-interviews.html">available here</ulink>
</para>
</sect1>
<html:hr/>
<sect1 role="toc">
<title>The interviews</title>
<sect2 >
<sect3 id="alcrooks">
<title>Alistair Crooks about the history, present state and future of pkgsrc</title>
<formalpara>
<title><emphasis>Hi Alistair, please introduce yourself</emphasis></title>
<blockquote>
<para>Ah, right, starting off with the difficult questions? I'm Alistair
Crooks, and live in the UK with my wife and children. I got involved
with BSD back in the 4.1c days (like 1985), installed 386BSD 0.0 on a
PC I had, moved to NetBSD 0.8 when it came about, became a NetBSD
developer in 1997 to set up pkgsrc, joined the NetBSD core team in
1999, was a director when the NetBSD Foundation was formed in 2001,
and have been President of the NetBSD Foundation since early
2005.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>First of all, congratulations to 10 years of pkgsrc! What are your
feelings?</emphasis></title>
<blockquote>
<para>I'm very pleasantly surprised, and thankful to all of the people who
have used pkgsrc, and contributed to its development, growth and
success.</para>
<para>I really didn't expect it to take off in quite such a spectacular
fashion, and I think a lot of that success can be laid at the feet of
the developers who steered it in the right direction over the last 10
years, and the people who used it, and told us what they liked, and
what they didn't like.</para>
<para>As with all large software systems, there are still things that we'd
like to change, and I hope that we'll get around to addressing some of
those soon.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>As a founder of pkgsrc ten years ago could you give us a brief
summary of the project's history?</emphasis></title>
<blockquote>
<para>pkgsrc was derived from the FreeBSD Ports system. NetBSD already
had a "port" in its naming scheme - that's a specific architecture or
platform to which NetBSD is ported - so NetBSD grew a packaging system
related to packages. We already had src, gnusrc, xsrc, and so we grew
a pkgsrc system, too - so the correct pronunciation is thus "package
source".</para>
<para>I imported the <firstterm>pkg_install</firstterm> routines first,
along with the main Makefile, and then I imported some packages piecemeal.
We then realized the power that such a system held for us (and, to be fair,
FreeBSD had realized this a while before), and the pkgsrc system's
depth grew as more and more packages were added.</para>
<para>In 1999, I was working at an investment bank in London, and needed a
packaging system to manage third-party software on a fairly large
network of Solaris 2.6 machines - pkgsrc fitted the bill, and so the
first non-NetBSD platform was Solaris, followed closely
by Linux. In 2001, I was asked to port pkgsrc to Mac OS X, and during that
exercise, it became apparent that we needed to get our act together
with portability. Up until that time, we'd been using a fairly heavy
compatibility layer which made every platform look like NetBSD -
Christos Zoulas developed it, and I called it <firstterm>Zoularis</firstterm>.
But around this time, it was apparent that we needed a more portable way of
doing this, and so I ported pkgsrc and its tools to a POSIX interface. The
rest of the 14 platforms of pkgsrc mainly followed from that.</para>
<para>I'm particularly happy with a number of the technical innovations we
made in pkgsrc - by checking for abstraction, rather than
implementation of the abstraction, we made pkgsrc a cleaner place to
work, and so we were able to port to more platforms easily -
scalability just happened. And early on, mainly because I was told it
wasn't possible, we implemented a consistent package numbering scheme
whereby we could tell if a package was older or newer than another
instance of that package. This has allowed us to provide auditing on
every pkgsrc system for vulnerable packages.</para>
<para>There are a number of technical innovations that Johnny C. Lam made in
pkgsrc itself, though, that make it stand out above all others. The
main one is, I think, Johnny's buildlink mechanism, whereby the
correct pre-requisite packages are used when building a package.
Before we had this system, if a machine had ncurses installed, for
example, ncurses was used in preference to the system curses when
building and linking. Or if the system already provided openssl
functionality, the packaging system would still build its own openssl
libraries, to make sure that the required functionality was available.
Johnny's buildlink system made sure that we could use pkgsrc or system
functionality, if preferred, along with sane defaults, and version
checking. It also made sure that we could link the package with the
desired libraries, rather than whatever was installed on the system at
the time. This allows us to perform bulk builds to make binary
packages available in a sane and easy manner.</para>
<para>Finally, others recognize the benefits of pkgsrc - Sun donated some
build equipment to the pkgsrc project some years ago, and the
Dragonfly BSD project have also adopted pkgsrc as their packaging
system, and provided some excellent developers to help us with this,
most notably Joerg Sonnenberger.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>What are the main benefits of the pkgsrc system?</emphasis></title>
<blockquote>
<para>Any packaging system should provide you with an easy way to install
third-party software. The BSD packaging systems, and
<firstterm>Gentoo's</firstterm>, provide
you with a way to do this from source, so that you can be sure that
the software that you install on your machines is the same as that
released by the package authors - it has not been modified in any way,
and that it is built and linked with the software that you have
previously built and linked, and by your compilers. It will also be
built and linked with the options that you chose.</para>
<para>pkgsrc itself is extremely portable, and its buildlink system is
unique. People point out to me that pkgsrc has only 7.300 packages ,
whilst others have many more. That's true, but</para>
<orderedlist numeration="loweralpha">
<listitem>
<para>that's 7.300 packages on multiple systems, so that if you are
suddenly gifted an HP/UX or Solaris or AIX system, and told to
administer it, you have an easy starting point</para>
</listitem>
<listitem>
<para>pkgsrc doesn't tend to split packages up into multiple -libs,
-docs, etc sub-packages like some other packaging systems, so the
package numbers are not artificially inflated,</para>
</listitem>
<listitem>
<para>we don't automatically target CPAN or CTAN, but prefer to
incorporate packages that people need, so that we don't accumulate
hubris in the system,</para>
</listitem>
<listitem>
<para>the pkgsrc developers tend to use pkgsrc for their own machines -
we drink our own champagne, and so any cruft is likely to be noticed
fairly soon, and suggestions made how to streamline things, and</para>
</listitem>
<listitem>
<para>pkgsrc has been able to sign binary packages (and verify the
signature of binary packages as part of the pkg_add process) but not
many people know about it. I see that this feature was added to pkgsrc
in late September 2001, so most people were otherwise occupied at the
time. This signature verification was used as part of the NetBSD
update facility that I talked about at Usenix in 2004, but NetBSD
update hasn't really progressed since then - that's more because I
haven't pushed this functionality rather than anything else.</para>
</listitem>
<listitem>
<para>pkgsrc has its own ecosystem, its breeding pool, called
<ulink url="http://pkgsrc-wip.sourceforge.net/"><firstterm>pkgsrc-wip</firstterm></ulink>
on sourceforge. It's a fantastic way for people new to packaging
systems can find their feet with the pkgsrc conventions, and also
an excellent way for pkgsrc to gain new packages, rather than
people being forced to send a problem report into NetBSD.</para>
</listitem>
</orderedlist>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Where and how do you use pkgsrc?</emphasis></title>
<blockquote>
<para>I have numerous machines at home upon which I run pkgsrc, some
non-NetBSD, some virtual. I use it in my work whenever I can. I'm a
strong believer that by using things whenever you can, these things
will succeed - people who use it are motivated to make it as efficient
and useful as possible. That's why pkgsrc has succeeded so well in
providing an extremely portable and useful third-party packaging
system.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>How does the pkgsrc release engineering work?</emphasis></title>
<blockquote>
<para>pkgsrc makes releases four times a year. Branches are cut at the end
of every quarter, and so each branch or release is named
pkgsrc-2007Q2, pkgsrc-2007Q3, named for the end of the quarter that
has just ended.</para>
<para>When a new release is made, sustaining engineering takes place on that
branch by the <emphasis>pkgsrc releng team</emphasis>.
The previous branch is deprecated.
Fixes for security vulnerabilities are pulled up to the branch. We
have been making releases like this now for a number of years, and
have found that it is just the right frequency.</para>
<para>Preparing for a release is done differently to NetBSD, and, I suspect,
to other projects - we institute a "freeze" for new functionality on
the HEAD, and enforce this ourselves. No new packages are allowed, and
changes to the infrastructure are severely discouraged. This takes two
weeks, and allows us to concentrate on the individual packages
themselves. We monitor the number of packages which fail to build,
install and deinstall properly by using bulk build results on a number
of platforms. These results are posted to the pkgsrc-bulk mailing
list, and differences from previous bulk build runs are also posted
there.</para>
<para>Having branched pkgsrc, sustaining engineering takes over - these guys
do a wonderful job looking after the branch, and making sure that
packages on the branch are kept as secure as possible. From me, and
anyone who uses the pkgsrc branches, to Lubomir Sedlacik and the rest
of the guys, a massive "thanks!".</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Apart from the release engineering, are there any other pkgsrc
projects you are currently working on?</emphasis></title>
<blockquote>
<para>I have just picked up an interesting project, along with Alan Barrett,
to bring <firstterm>system packages</firstterm> to NetBSD - that is a
little bit pkgsrc-related. The situation that we have now is that the system
packages are able to be built from NetBSD's <firstterm>build.sh</firstterm>,
but we lack the ability to install them properly - right now, our pkg_install tools do
not allow us to install into a <envar>${DESTDIR}</envar> from the command line,
and so there is some work under way in that area.</para>
<para>I have a list of other projects which I'd like to see implemented,
such as
<orderedlist numeration="loweralpha">
<listitem><para>finishing off the pkg_install rewrite</para></listitem>
<listitem><para>finishing off DESTDIR builds</para></listitem>
<listitem><para>an implementation of pkgviews</para></listitem>
<listitem><para>documenting pkgsrc for users a bit better</para></listitem>
<listitem><para>some high-level tools, such as some of the Linux distributions have, for using and
managing binary packages, for those of us who prefer not to use source and to
trust others compilation systems (again, see the digitally-signed binary
discussion earlier), and</para></listitem>
<listitem><para>I have some long-term research I'd like to do using the
<emphasis>fanout file system</emphasis> which I
committed as part of the refuse work, which implements a specialized union file
system using the <firstterm>ReFUSE</firstterm> library, and which could be
used to great effect to provide package addition in a controlled, transaction-based
fashion.</para></listitem>
</orderedlist>
</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>If you analyse the current state of pkgsrc, which improvements and
changes do you wish for the future?</emphasis></title>
<blockquote>
<para>I'm very happy with the current state of pkgsrc, and look forward to
some of the projects which are in progress coming to fruition. Adrian
Portelli's rewrite of the audit-packages utility in C brings some
welcome speed to the process of keeping our systems secure, Joerg's
rewrite of pkg_install is coming along slowly but nicely, and will be
a great benefit. The whole pkgsrc infrastructure has been broken out
of one massive Makefile into a number of snippets, and that has made a
difference in terms of speed, I suspect. We also need some good
documentation on all aspects of pkgsrc - our documentation is good, but
we could always do with some more - and every so often, I think it
would be nice to rationalize our directory hierarchy to make it easier
to find things. However, nowadays, if I want to find out if pkgsrc has
an entry for a piece of software, the good folks at
<ulink url="http://pkgsrc.se/">pkgsrc.se</ulink> provide
an excellent web-based search engine, which also covers the pkgsrc-wip
project. I'd also like to see more packages move from pkgsrc-wip to
pkgsrc, and for us to take advantage of some of the talent out there
that provided all the entries in pkgsrc-wip.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>On this years pkgsrcCon
in Barcelona (<ulink url="http://www.pkgsrccon.org/2007/">pkgsrcCon 2007</ulink>),
Christoph Badura talked about problems with pkgsrc for ordinary users. So is pkgsrc great
for developers and system builders but to heavy to use for
ordinary users?</emphasis></title>
<blockquote>
<para>I wasn't at pkgsrcCon in Barcelona, but I've read Christoph's
presentation. The message I took out of it wasn't that pkgsrc is only
for the tech-savvy (it's not), but that pkgsrc is used by a wide
variety of people, and that there would be gains to be made if we were
to structure a higher-level tool on top of pkgsrc which could deal
with some of the higher level issues.</para>
<para>I fully support Christoph's views on things - I've known him a long
time, and he and I are great friends - and I'd like to think that
we've already addressed some of the issues that he talks about, and
that we're thinking about the others.</para>
<para>I see pkgsrc being used by anyone who can type
<userinput>make install</userinput> -
definitely not just the technically-aware people. I also think we
could do with a high-level front-end for the pkgsrc system, so that I
could just type "pkg x11-drm" and it would do the necessary magic
itself, and provide you with a DRI-enabled X11 setup. That's one area
we can learn from the Linux distributions (which, with the exception
of Gentoo, almost exclusively use binary packages). But remember that
we can already add binary packages just by giving pkg_add an URL - it's
just that some people prefer to compile things themselves, and that
the binary package route isn't to everyone's tastes.</para>
<para>I'm a firm believer in learning from what other people do - at the
recent <firstterm>EuroBSDcon</firstterm> in Copenhagen, I learned
that Gentoo's packaging system has a <emphasis>--pretend</emphasis>
argument, which simply
tells you the actions it will carry out, without actually doing them. I often
find myself thinking "would that be useful for pkgsrc to have?", and, in this
case, I think it would - there are a number of times when I come to a
machine and I wonder what vintage the gtk2 package is, or whether I am
about to tie up a box for hours of compilation if I try to upgrade a
package now - pkgsrc has a <firstterm>pkg_rolling-replace</firstterm>
which will run through all of a package's pre-requisites, working out whether
the pre-req needs to be updated or not, and doing an update in place, if it
does. Incredibly useful functionality; occasionally, it ties up the machine
for a while.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>There was also a presentation about the
pkgjam project, a "radically re-engineered" version of pkgsrc,
in development by James K. Lowden and Johnny C. Lam.
What do you think about it?</emphasis></title>
<blockquote>
<para>It's an
interesting piece of research. In some ways it addresses the
implication behind your previous question, but there are also elements
of package views about pkgjam which I find interesting, too. I don't
see pkgjam as a competitor to pkgsrc - I see it as adding something to
pkgsrc, of being complementary, and of addressing some different areas
to the core pkgsrc ones.</para>
<para>A lot of the Linux distributions have low-level packaging tools, and
higher-level tools built on top of them, which are able to work out
pre-requisite packages, and queue them up to be downloaded, etc. I'd
love to see pkgjam or some other system do the same for pkgsrc.</para>
<para><emphasis>[See also the
<link linkend="lowden">interview with James K. Lowden</link>
for more details about pkgjam]</emphasis></para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Do you have any practical tips to share with the
pkgsrc users?</emphasis></title>
<blockquote>
<para>Sure.</para>
</blockquote>
<blockquote>
<itemizedlist>
<listitem>
<para>I always find it best to remember my wife's birthday.</para>
</listitem>
<listitem>
<para>The <ulink url="http://www.pkgsrc.se">pkgsrc.se</ulink> search
facility, which covers both pkgsrc and pkgsrc-wip, is very, very
useful. If you can't see a package entry in pkgsrc, try pkgsrc.se - it may be
under a different name, or in an unusual category, and their searching
and indexing facility, is very, very good.</para>
</listitem>
<listitem>
<para>I like to build up new package sets in a
<ulink url="http://pkgsrc.se/pkgtools/pkg_comp">pkg_comp</ulink>
chroot, and then deploy en masse. I find that if I do this when I build new NetBSD
sets, then I have consistent packages across my machines. This may not
suit everyone's way of working, but I find that this suits my needs
very well. There is a fairly high bar to start using pkg_comp, but
it's well worth it.</para>
</listitem>
<listitem>
<para>As far as useful packages go, I use the usual suspects fairly
extensively - <firstterm>screen</firstterm>,
<firstterm>xcb</firstterm>, and a number of editors and
environments which aren't really interesting for anyone else. I have
some personal packages which I carry around, and which I use for
setting consistent environments on all types of machines.
I'd make them available, but I doubt anyone would be interested in
them, as they're highly environment-dependent. That's my
environment.</para>
</listitem>
</itemizedlist>
</blockquote>
<para>
<emphasis><emphasis role="bold">Thank you very much for
taking the time to answer the questions!</emphasis></emphasis>
</para>
</formalpara>
<note>
<title>Source-based package building systems on Linux distributions</title>
<para><ulink url="http://www.gentoo.org/">Gentoo </ulink>is not the only
Linux distribution with a source-based packaging system. Other Linux projects
have adopted the concept as well:
<itemizedlist>
<listitem><para><ulink url="http://sorcerer.aakin.net">Sorcerer</ulink></para></listitem>
<listitem><para> <ulink url="http://www.sourcemage.org">Source Mage</ulink></para></listitem>
<listitem><para><ulink url="http://www.lunar-linux.org">Lunar Linux</ulink></para></listitem>
<listitem><para><ulink url="http://crux.nu/">CRUX</ulink></para></listitem>
<listitem><para><ulink url="http://www.rocklinux.org/"> ROCK Linux</ulink></para></listitem>
<listitem><para><ulink url="http://voltalinux.sicurezzarete.com/">Voltalinux</ulink> is a Linux distribution based
on Slackware and pkgsrc.</para></listitem>
</itemizedlist>
</para>
</note>
</sect3>
</sect2>
<html:hr/>
<sect2>
<sect3 id="feyrer">
<title>Hubert Feyrer: The history, failures and prospects of pkgsrc</title>
<formalpara>
<title><emphasis>Hi Hubert, please introduce yourself</emphasis></title>
</formalpara>
<blockquote>
<para>I'm currently living in Regensburg,
which is in southern Germany. The fact that I'm from Bavaria but don't
like beer (and wine) always puzzles people, but well, that's me. I've
studied Computer Science, and used NetBSD from its very early days on
the <emphasis>Amiga</emphasis>, ran the master FTP site for
<ulink url="http://www.NetBSD.org/ports/amiga/"> NetBSD/amiga</ulink>
for many years starting in 1993, and became a NetBSD developer in 1997.
The NetBSD/amiga site had a strong focus on providing precompiled
binary packages, I have written my own packaging system back then,
and when NetBSD decided to get some "official" packaging system, I
was happy to help out. As a funny side note, the official reason for
my developer account was that I've sent a slew of problem reports
back then after I installed NetBSD/i386 for the first time, and my
official working area always was "misc bug fixing". ;).
<emphasis> [<ulink url="http://www.feyrer.de/lebenslauf/netbsd.html">Hubert's NetBSD CV</ulink>]</emphasis>
</para>
</blockquote>
<formalpara>
<title><emphasis>10 years of pkgsrc! What are your
feelings?</emphasis></title>
<blockquote>
<para>"Wow, it still lives!'' - my feelings are mixed about it. On
the one hand, I'm massively impressed by what pkgsrc is today:
cross platform, providing many packages, on a number of non-NetBSD
platforms that I use (Debian Linux, Mac OS X, Solaris). On the other
hand, the original goal of providing binary packages for NetBSD is
not really in sight, and doesn't seem to be as much on focus as I
wanted. pkgsrc has evolved from the
<emphasis>NetBSD Packages Collection</emphasis> to a
much wider focus, and I guess that's the consequence -- a win for
many, but with some drawbacks for the original system.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>As a founder of pkgsrc ten years ago could you give us an
overview of the projects history?</emphasis></title>
<blockquote>
<para> I've already told you what happened for NetBSD/amiga above.
NetBSD's core group decided to get a packaging system, and when I met
<personname>Jason Thorpe</personname> (who was in the
<emphasis>NetBSD Core Team</emphasis> then) during the Munich
<orgname>IETF</orgname> meeting in 1997, it seems that I've provided
some input that were good enough to get me on board. Besides me,
Alistair Crooks was in the game. He had experience with the
<emphasis>FreeBSD Ports system,</emphasis> and
also worked on software packaging for other Unix systems. A third
person was on the list of people to work on, but that guy never
showed up.</para>
<para>First steps were getting the build system going, adapting it to
NetBSD and the various hardware platforms, and then starting to get a
few packages compiled. In the start we tried to keep things in sync
with the FreeBSD's ports system, but that didn't work out somehow,
and we forked off pkgsrc. If someone wonders: the NetBSD CVS
repository was split up into a few chunks back then, so that the
<filename>cryptosrc</filename> could be split out between the USA and
the rest of the free world. Other parts were
<filename>gnusrc</filename>, <filename>xsrc</filename> (which is
still there today) and probably others. So "pkgsrc" did fit into the naming
scheme. Most of the other parts were merged again later.</para>
<para>After running our own show, Alistair and I started to realize that we
need more manpower. We wondered whom to fetch in the boat, and
<personname>Matthias 'tron' Scheler</personname> was the first one.
I knew Matthias from the Amiga scene, and as he had a lot of Unix
knowledge and NetBSD enthusiasm, we made the right choice in having
him join pkgsrc. Many others followed, and I've since lost overview of
what people are working on pkgsrc - there are more than 80 developers
on the internal packages developers list today.</para>
<para>Milestones that I participated in pkgsrc were adding
<emphasis> wildcards</emphasis> for handling package versions, the
pkgsrc guide, and the bulk build framework. An article about the former is
<ulink url="http://www.NetBSD.org/docs/software/pkg-wildcards.html">available</ulink>.
</para>
<para>
A major focus in my work back
then was to make sure that wildcards also works for installing binary packages,
either from local storage or over FTP. The pkgsrc guide is less maintained as it
used to be today, but still
<ulink url="http://www.NetBSD.org/docs/pkgsrc/index.html">available</ulink>.
The bulk build framework was developed during the NetBSD 1.3 release cycle, when
Alistair and I made sure that packages did build. I needed a tool that allowed me to
verify that all packages that were depending on a changed package still build or
were rebuilt (i.e. that updated binary packages were available -- remember my
original goal to make binary packages available!), and as none existed
I made it. The bulk build framework is still in use today; An
alternative was made only recently by Joerg Sonnenberger, but I don't
know about it. "My" bulk build framework was later heavily pounded on
by Dan McMahill and others, and
documentation on its use can be found
<ulink url="http://www.NetBSD.org/docs/pkgsrc/bulk.html">here</ulink>.
</para>
<para>The management structure of pkgsrc was very shallow by then,
with Alistair Crooks doing most of the
"administrative" tasks, and developers picking up items that they felt
needed doing. One issue that that started to come up in pkgsrc
maintenance was security. Of course software has problems, and fixing
them is important. pkgsrc grew some people who adopted pkgsrc security
as their working area. This is still in effect today, and a lot of effort goes
into the stable branches that were introduced later on, to get those security
updates for people who want to use an otherwise stable version of
pkgsrc.</para>
<para>Personally, I've started doing marketing, advocacy and
documentation for the NetBSD operating systems by then, after pkgsrc
started to run on its own. I still ran the bulk builds for NetBSD/i386
back then, but after unclear policies on how to handle security
updates with respect to binary packages
(<ulink url="http://www.feyrer.de/NetBSD/blog.html/nb_20050411_0020.html">link</ulink>),
I've passed that hat on. That was more than two years ago, in 2005.
I admit that I haven't followed a lot of pkgsrc development since
then.</para>
<para>From my personal perspective today, items that I think are
lacking are:</para>
<orderedlist>
<listitem>
<para>making binary packages available for as many platforms
as possible, and</para>
</listitem>
<listitem>
<para>upgrading of installed packages, either by compiling or from
a set of precompiled binary packages. Crosscompiling is probably
of help there, and with some
<ulink url=" http://netbsd-soc.sourceforge.net/projects/xpkgsrc/">recent development</ulink>
this may actually change, eventually.</para>
</listitem>
</orderedlist>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>What were milestones of the project?</emphasis></title>
<blockquote>
<itemizedlist>
<listitem><para>~1997 import from FreeBSD</para></listitem>
<listitem><para>decide that we fork pkgsrc</para></listitem>
<listitem><para>first pkgsrc developers besides Alistair Crooks and me</para></listitem>
<listitem><para>1998 package wildcards</para></listitem>
<listitem><para>1999 bulk build framework</para></listitem>
<listitem><para>2001 pkgsrc agreed as base for the Open Packages
project, which aimed at making 1 packages collection for all BSDs.
The missing parts were adding whatever
FreeBSD and OpenBSD offered
at the time to pkgsrc to have the best of all worlds, but it never
happened.</para>
</listitem>
</itemizedlist>
<para>I've missed much of the more recent developments, I think it's
better to ask Alistair Crooks for them. He's still working on pkgsrc,
I consider myself out of that game for some years now, really.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Where and how do you use pkgsrc?</emphasis></title>
<blockquote>
<para>Systems that I use pkgsrc on currently:</para>
<itemizedlist>
<listitem>
<para>The <emphasis>Virtual Unix Lab</emphasis>
is based on NetBSD, and packages are installed
via pkgsrc, of course.
That's <ulink url="http://www.feyrer.de/vulab">my PhD project</ulink>
that I'm spending most of my time on right now.</para>
</listitem>
<listitem>
<para>I give a lecture on "Open Source" at the
<firstterm>University of Applied Sciences Regensburg</firstterm> and
when talking about source-based packaging systems I show pkgsrc as
example.</para>
</listitem>
<listitem><para> I've moved my mail setup to a friend's Vserver Debian Linux box,
and on that that I use pkgsrc without root privileges to install related
software: alpine, spamassassin, spamprobe, procmail.</para>
</listitem>
<listitem><para>I've replaced my desktop machine at home a few months ago. It used to
be a PC running NetBSD, it's a MacMini running Mac OS X now. pkgsrc
there is a huge help to give me the working environment I'm used to.</para>
</listitem>
</itemizedlist>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>What are the benefits of the pkgsrc system?</emphasis></title>
<blockquote>
<para>It's portable across many operating systems and hardware
platforms. This makes it really unique. Of course if you know a system
from the inside somewhat, you also know a lot of the issues, and I
could probably talk a lot more about the drawbacks than about the
benefits, but I've done so enough above already. ;)</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>If you analyse the current state of pkgsrc, which improvements
and changes do you wish for the future?</emphasis></title>
<blockquote>
<para> See above: availability of binary packages and updating of
installed packages. Documentation. And one thing that I've stumbled
across a lot recently: stability. Stable branches don't cut it for latest
versions, and pkgsrc-current is an ever-changing base system that
has failed many times recently. I hope there will be some convergence
at some near point in the future. (But I'm observing pkgsrc for some
years now, waiting to see that convergence...)</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>In 2005 Alistair Crooks gave a talk about the implementation of
pkgsrc. He said, that
<ulink url="http://www.pkgsrccon.org/2005/slides/no.pdf">he regrets nothing</ulink>.
What would you make different today, if you had to start over again?</emphasis></title>
<blockquote>
<para> Not much... maybe a bit more "steering" and goal-setting,
making it clear what we do and do not want. The focus between what
pkgsrc is/does today, and what NetBSD needs is different/conflicting
there.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Last question: Do you have any practical tips to share with the
pkgsrc users?</emphasis></title>
<blockquote>
<para>Contribute!</para>
<para>Besides work on the infrastructure (pkgsrc/mk/*) and packages, a
lot of other work is to do, and this is of equal importance:
Advocacy.</para>
<itemizedlist>
<listitem><para>write articles in papers</para></listitem>
<listitem><para>talk about pkgsrc on your pet platform at conferences</para></listitem>
* <listitem><para>update the pkgsrc guide</para></listitem>
<listitem><para>go to conferences and trade shows, and show people where
pkgsrc is today, and that it's of real use!</para></listitem>
</itemizedlist>
</blockquote>
</formalpara>
<para><emphasis><emphasis role="bold">Thank you very much for taking the time to
answer the questions!</emphasis></emphasis></para>
</sect3>
</sect2>
<html:hr/>
<sect2>
<sect3 id="hubbard">
<title>Jordan K. Hubbard: The invention of the FreeBSD Ports system,
its drawbacks and the merits of MacPorts</title>
<formalpara>
<title><emphasis>Hi Jordan, what are you currently
doing?</emphasis></title>
<blockquote>
<para>I work for <firstterm>Apple, Inc.</firstterm>, managing several
groups who are responsible for various bits of the "core OS" portions
of <emphasis>Mac OS X</emphasis>. This includes a lot of the open
source portions of it.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Do you still follow the development of the BSD systems?
Is there anything you like especially?</emphasis></title>
<blockquote>
<para>I do try to follow what the various groups are doing, though I
can't say I'm always as up to date on that as I would like to be. I
still have a strong sentimental attachment to FreeBSD, for obvious
reasons, though all the *BSDs are definitely still doing good work
and BSD "satellite projects" like
<ulink url="http://www.openssh.org"><emphasis>OpenSSH</emphasis></ulink>
have clearly made a very positive impact in their own right. If I
had to say that I liked anything "especially" about the BSD systems,
it would have to be their level of integration (not just from the
"consumer" level, but as a developer trying to fetch, modify and
build all the necessary bits without having to first go on an
easter-egg hunt) and, of course, the liberal and easily understood
BSD license.</para>
</blockquote>
</formalpara>
<formalpara>
<title> <emphasis>Please tell us about your ideas to implement the
FreeBSD Ports system. How did it all started?</emphasis></title>
<blockquote>
<para>As much as I'd like to say that it all suddenly came to me in a
blinding flash of inspiration one day, the truth is somewhat more
prosaic. I was simply getting more and more frustrated with the fact
that every new FreeBSD system I built and installed (and I was
building quite a few PCs in those days) needed to be customized with
pretty much the same "extra bits" every time (bash, emacs, MH, and so
on) and every time was also basically the same old drill - remember
where to find the bits, fetch and unpack the bits, configure/patch
the bits as necessary, build and install. As some folks will remember,
FreeBSD also didn't have a binary package management system back then
(I wrote pkg_install sometime after the ports collection) so you had to
build all the add-ons yourself by hand.</para>
<para>Any repetitive task like this positively screams "automate me!"
to a software engineer, of course, and given that I was pretty handy
with Berkeley <emphasis>make(1)</emphasis>, it didn't take me long
to quickly whip up what became <emphasis>bsd.port.mk</emphasis>
and create a couple of dozen "ports" to test and refine the concept.
In a couple of weeks I had all my favorite bits of software under the new
system and, in August of 1994, I felt it had reached the stage where it
would be useful enough to others that I checked it into the FreeBSD CVS
repository. It proved popular enough that, with the help of a number
of other folks, I quickly got the collection (and bsd.port.mk refined)
to the point where we had several hundred ports and a growing
conviction that we had created something pretty useful.</para>
<para>By then it was also becoming clear to me that we were going to
need some sort of package management system which allowed end-users
to have all the benefits of "make install" in the ports collection without
having to actually go through all the intervening steps, so I dragooned
<personname>Satoshi Asami</personname> (who had proved himself
to be a highly capable and productive creator of ports) into the role of
<emphasis>Ports Meister</emphasis> and went off to write
<emphasis>pkg_install(1)</emphasis> as the logical counterpart to ports.
The rest, as they say, is history.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>You invented the Ports system in 1994. Ports-like systems are
still very popular, there was even a boom on new Linux projects.
Why are the source-based package management systems
so popular?</emphasis></title>
<blockquote>
<para>Because they're so necessary? :-)</para>
<para>Any significant collection of binary packages has to come from
somewhere, obviously, and the only real decision to be made is whether
you want to hide the machinery in the sausage factory or polish it up to
the extent that you're comfortable with other people seeing it (and using
it to make their own sausages). I'm sure there are more attractive analogies,
but the process of keeping lots of software up to date and continuously
impedance-matched to your target platform is always an ugly, labor intensive
process at best so it's a reasonably accurate one.</para>
<para>You also have to consider the fact that both the add-on software
and the target platforms are slowly but constantly changing so your
"labor costs" are ongoing - you can't just build the sausage factory and
then walk away and trust the automation to take care of itself. This
means you can either pay a small army of release engineers to manage
the process, as the commercial OSes do, or you can expose the inner
workings and rely on volunteers to help you keep the machinery
in order.</para>
<para>From a pure engineering standpoint, source-based package
management also has a number of advantages that you just can't get with
static binary package management systems, the most obvious and significant
being the degree to which you can customize the software during the process
of building it. A lot of software also simply cannot be customized any other way
(a somewhat lazy decision for a developer to make, but hardly rare).</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Did you followed the development of pkgsrc?
Anything you liked about it?</emphasis></title>
<blockquote>
<para>I haven't followed it all that closely, to be honest, but my initial inspection of
it back when it first came out left me with the impression that they'd taken a number
of my ideas and run further with them, cleaning up both the design and implementation
of the system to a fair extent.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Are you still working on ports-like systems?</emphasis></title>
<blockquote>
<para> I helped start <firstterm>MacPorts</firstterm>
(and argued many aspects of its design with the primary
implementor(s)) so I guess that'd be a yes, though I did a lot
less of the engineering work this time around.
I'm in management now. :-)</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>What's about the
<ulink url="http://www.macports.org/">MacPorts project</ulink>?
How is it different and better compared to the FreeBSD Ports
or pkgsrc?</emphasis></title>
<blockquote>
<para>Well, one of the mistakes I made with FreeBSD ports was using in
make(1). That is a fine tool for creating "linear recipes" for doing things, don't get me wrong -
that's why I used it. What became apparent as the ports collection evolved, however, was
that the Makefile syntax was clearly less flexible and powerful than, say, an actual programming
language and also less than easy to mine data out of or apply batch changes to as the system
evolved and chunks of oft-duplicated "code" got swept into the base system itself.</para>
<para>MacPorts uses an actual scripting language, namely <firstterm>TCL</firstterm>,
in creating port descriptions. This gives the port writer a lot more flexibility in dealing with
some of the more complex bits of software out there and the whole notion of action hooks
(pre/post fetch, build, install, etc) is much cleaner in MacPorts, as is the notion of versioning
the MacPorts infrastructure itself such that it can substantially evolve without breaking old port
descriptions (a port description can "link" with a specific
version of the MacPorts system, much as a dynamic library can have multiple versions of itself
installed simultaneously). I'm not sure we solved the data mining and batch modification problem,
per-se, but using XML as a description format had its own problems (it was something we tried early
on) so I'm not sure the trade-offs for that would have been worth it in any case.</para>
<para>I'm not enough of an expert with pkgsrc to really do a comparison between it and MacPorts
justice, but I'll simply say that MacPorts learned a lot from the systems that had come before and
tried a different (though nonetheless parallel) approach. Whether or not the end result is "better" is a
highly subjective question and I'll leave that for the users to decide.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>What are the problems of ports-like systems?
What do you think about their future?</emphasis></title>
<blockquote>
<para>I think the biggest problem all ports systems face is scaling. As the
number of port descriptions increase, the maintenance costs go up
proportionately (some might even say exponentially, in some of the systems out
there :-) . More ports means more testing burden, more volunteers to manage
them, more version skew across the collection as a whole, and the number of
broken ports as a percentage of the whole collection can get quickly out of
hand without some pretty rigorous process automation. Think of it like raising
sheep: 3 or 4 sheep are relatively easy to take care of and almost any 4H
student can manage it, but 10,000 sheep is another matter entirely - it takes
someone with real skills in animal (or software) husbandry to cope with that
many successfully and not simply have thousands of dead sheep, errr, ports in a
year's time.</para>
<para>I also think that we have to increasingly consider the end-user impact of
having that much software potentially added to one's system. Most software
added via a well designed/maintained port or package management system is
reasonably well-behaved (often because it's forced to be, through automated
checks for unintentional overlaps / conflicts) but even the most professionally
run ports collections miss odd corner case conflicts or fail to test every
possible combination of software. I'd like to see some new ideas in the future
about how to manage the filesystem namespace and make it easier for
packages to be grouped together into "views" of some sort, where the
filesystem hierarchy the user saw might depend on the software they wanted
to run at any particularly time. Individual processes could also have their own
views, preventing unintended behavior in packages which sometimes attempt
to be too clever in enabling optional (and possibly unwanted) features when
other packages are detected.</para>
<para>This is never more evident than it is during build time, when the popular
autoconf system (which was designed with the needs of a different era of
computing in mind) sniffs around your system and finds things you may or may
not want it to. Building software in very carefully controlled "sandboxes" is
clearly necessary for creating reproducible builds (and controlling
dependencies) but there is still no runtime equivalent. We may be a ways off
from truly needing one, but it's still a problem worth thinking about while
talking "futures."</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Last question: What would you make different today, if you had to
start over again (package management on FreeBSD with more than 17.000
packages)?</emphasis></title>
<blockquote>
<para>That's a tough question. Clearly, the FreeBSD ports collection has been
highly successful - 17.000 is an almost staggering number of ports - so they've
clearly gotten around the structural limitations of make(1) and it's even a
plausible argument that using something which pretty much everyone already
understood was a significant part of that success. I know that requiring people
to learn at least some TCL as a prerequisite to joining the MacPorts project has
been a bit off-putting to some and the barrier to entry is certainly higher
there.</para>
<para>The package management system is an easier call - I wrote that very
quickly under the assumption that we'd throw it away relatively soon once we
found a better one, but somehow that never happened. Starting over today,
I'd look at source code management systems like
<emphasis>subversion</emphasis> or <emphasis>git</emphasis> as a
better conceptual model for creating and permuting what's currently thought of as
"the binary base system." Installation would basically consist of selecting
one or more tags (which could be branches) and saying "please make my
system look like this", the appropriate bits then flowing over the wire and
getting laid down on disk. This would make it very easy to create customized
versions of the OS for specific needs, or just for experimental purposes, and
all the flexibility that <emphasis>SCM</emphasis> gives you for code
development could be applied to the OS and package installation process.
This isn't even a particularly new idea - systems like
<emphasis>Conary</emphasis> (started by some of the guys behind RPM)
have been doing this for awhile in the Linux space.</para>
<para>Of course, if I wanted to get really radical, I would also challenge Unix's
fundamental notion that you can have any filesystem namespace organization
you like, just so long as it's hierarchical and system global, since things like
search and package management are really penalized by that notion if you
have any desire to "do them right", but that's probably beyond the scope of
what most *BSD projects would even care to grapple with.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Thank you very much for taking the time to answer the
questions!</emphasis></title>
<blockquote>
<para>My pleasure! I hope this helps to establish some historical context for
how things came to be the way they are today.</para>
</blockquote>
</formalpara>
<note>
<title>Recommendation: more interviews about pkgsrc</title>
<itemizedlist>
<listitem>
<para>NetBSD and pkgsrc developer Jeremy Reed published in 2006 an
interview with non-NetBSD pkgsrc users:
<ulink url="http://www.bsdnewsletter.com/2006/10/Features174.html">"Pkgsrc on non-NetBSD interview"</ulink>
</para>
</listitem>
<listitem>
<para>pkgsrc developer Johnny Lam gave an audio interview about the
merits and drawbacks of pkgsrc. He talked also about the pkgsrcCon
conference. The <ulink url="http://cisx1.uma.maine.edu/~wbackman/bsdtalk/bsdtalk083.mp3">podcast</ulink>
is available on <firstterm>bsdtalk</firstterm>
(<ulink url="http://bsdtalk.blogspot.com">http://bsdtalk.blogspot.com</ulink>)
</para>
</listitem>
</itemizedlist>
</note>
</sect3>
</sect2>
<html:hr/>
<sect2>
<sect3 id="freebsd">
<title>Erwin Lansing about the FreeBSD Ports</title>
<formalpara>
<title><emphasis>Hi Erwin, please introduce yourself</emphasis></title>
<blockquote>
<para>
I have been a FreeBSD ports committer since 2003 and joined the portmgr
team in 2005.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>The 10th anniversary of pkgsrc seems to be an
impressive affirmation of the ports concept, invented on FreeBSD in
1994. What do you think?</emphasis></title>
<blockquote>
<para>Indeed it is. Not only is the concept thriving both in FreeBSD ports
and pkgsrc, it has also bee adapted by e.g. <firstterm>Gentoo portage</firstterm>,
clearly showing that this a strong concept, that scales not only in number of
applications, but also across different operating systems and platforms.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Which role does the pkgsrc play as an inspiration for the developers
of the FreeBSD ports system?</emphasis></title>
<blockquote>
<para>My impression from talking to pkgsrc developers at conferences, is that
there is a lot of activity going on in the architecture of pkgsrc
itself. While FreeBSD ports outdoes pkgsrc in numbers, pkgsrc shines in
features. FreeBSD developers certainly can get inspiration on new feature to
implement in FreeBSD port, but you can imagine that it is not a
trivial task to do so with over 17.000 ports, each of which can break in
new and interesting ways.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>One outstanding feature of the FreeBSD ports system is the
amount of buildable packages - more than 17000 are currently available! A few
years ago, Jordan K. Hubbard, the inventor of the ports system,
predicted, that the system would become very heavy to maintain with more
than 10000 packages. It seems the FreeBSD ports developers and
maintainers have proved him wrong. How was this problem
solved?</emphasis></title>
<blockquote>
<para>Blood, sweat and tears. And did I mention: testing, testing, testing?
We can thank a large number of our dedicated users and committers for
adding all these packages and keeping them up-to-date. However, keeping
high quality on such a vast number of packages could not have been
achieved without many hours of testing whether they can build, install
and deinstall by the portmgr team, especially Kris Kennaway.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Thank you very much for taking the time to
answer the questions!</emphasis></title>
<blockquote>
<para>Thanks, and congratulations to pkgsrc!</para>
</blockquote>
</formalpara>
</sect3>
</sect2>
<html:hr/>
<sect2>
<sect3 id="espie">
<title>Marc Espie: Comparison of pkgsrc and the OpenBSD Ports system</title>
<formalpara>
<title><emphasis>Hi Marc, please introduce yourself</emphasis></title>
<blockquote>
<para>
[I'm] living in Paris, France, and liking that town.
I am currently a teacher/advisor for students at a computing and managing
school.</para>
<para>I've been part of the OpenBSD project for about 10 years now. I started
with small corrections and well... these things grow. Since we have a
culture of fixing things that don't work like we want, I ended up being
responsible for our <firstterm>make</firstterm>, our
<firstterm>m4</firstterm>, the redesign of our ports tree infrastructure
and the complete rewrite of our pkgtools.... among other things.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Which role does pkgsrc play as an inspiration for your work on
OpenBSD Ports?</emphasis></title>
<blockquote>
<para>I use it as a source of inspiration. I read the corresponding mailing-lists,
and I look at what you guys add to it, taking ideas from it. It's probably
no longer possible for us to take actual code, since we have diverged so
much. But I've read about your update ideas, the
<ulink url="http://www.NetBSD.org/docs/software/pkgviews.pdf"><firstterm>pkgviews</firstterm></ulink>
stuff, and the <firstterm>buildlink</firstterm> stuff.</para>
<para> I think that the last idea I lifted in a visible manner was the
<emphasis>xpkgwedge</emphasis>, which was a very nice idea.</para>
<para>Actually, it is a very important source for me. Writing code is more or
less trivial. Having the right design ideas is harder. So, taking code,
extracting the gems in it, and re-injecting it in our code makes it better.
Not being able to lift the code straight away is also a good thing actually,
because I cannot be lazy, I have to think about it, and get only the good
parts. I can leave the clunky stuff behind. ;)</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>What do you like about pkgsrc?</emphasis></title>
<blockquote>
<para>Well, the fact that it exists. It's good to have competition. And also a
kind of biodiversity. I can see some of your experiments, how they turn
out, the ones that work out, and the ones that don't work out.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>What are the main differences between pkgsrc
and OpenBSD ports?</emphasis></title>
<blockquote>
<para>I think we're better. ;)
No, kidding aside, we have totally different design goals, so it's
quite normal we have very different implementations, even though we
tend to converge on ideas we share. If I judge the OpenBSD ports
by the design goals we have, it is better than pkgsrc (which is good
for my sanity), but there are things we do not even try to do.</para>
<para>Our main goals are binary packages, and reproducibility. With our
paranoia, we don't like endless options that make it hard to reproduce
issues, so we have a lot less configurability and portability than pkgsrc.
It would probably be possible to take the OpenBSD framework and make it
run on something else, but that's not a goal, so you would probably have
to take a lot of OpenBSD with you. We also have just a few knobs.</para>
<para>Some specific points about us:</para>
<itemizedlist>
<listitem><para>we made definite choices about our toolset, and we
decided to base the binary tools on <emphasis>Perl</emphasis>. As a
result we now have an extensive collection of tools that work all together.
Instead of snippets of code in sed/python/ruby/whatever that do the same
job again and again, we have ONE single routine that can read and write
packing-lists, and a standard simple way to manipulate that. So, any
addition to packing-lists syntax can be done in one single location
(the packing-list entities all inherit from OpenBSD::PackingElement, so you
don't even have to write new behavior if you don't need it).</para></listitem>
<listitem><para>we have focused on reproducible binary packages over five years ago.
We developed the fake framework at that point, and I tweaked our make
to be more POSIX-compliant (any variable passed on the command-line ends
up in sub-makes through .MAKEFLAGS. Case in point: DESTDIR). As a result,
the install stage does not work directly. We do in in three steps:
<orderedlist>
<listitem><para><userinput>make fake</userinput> installs the package in a separate tree</para></listitem>
<listitem><para><userinput>make package</userinput> builds the binary package</para></listitem>
<listitem><para><userinput>make install</userinput> just calls pkg_add</para></listitem>
</orderedlist>
So we are certain that binary packages are tested, because they get installed
all the time. This is also very good to catch mistakes in packing-lists.</para></listitem>
<listitem><para>we removed a lot of extra luggage in
<emphasis>bsd.port.mk</emphasis>. It was easier for us
when we did it, because we were a lot smaller (each mechanism you remove,
you need to sweep the tree to kill it first, which is very hard to do
when you have >4.000 ports). I've also made systematic use of shell fragments:
anything that could be duplicated ends up in an internal variable. There's
also a complete manpage that documents over 99% of bsd.port.mk visible
variables, instead of having them in comments.
I've noticed NetBSD has finally cut up the huge
<emphasis>pkgsrc.mk</emphasis> into small pieces,
which is good. I'll admit to not having looked at your documentation
recently. I think we should steal a big chunk of the <emphasis>FreeBSD porting
handbook</emphasis>, and some of the NetBSD supplements. ;)</para></listitem>
<listitem><para>we have an integrated mechanism to handle options (FLAVORS) and to
be able to build several packages from one single port (MULTI_PACKAGES).
This was fairly hard to get to work as it does today, but it is very
useful.</para></listitem>
<listitem><para>we're a bit farther along the road where semi-automatic
updates are concerned. Mostly by having one single way of handling it, instead of
trying several approaches.</para></listitem>
</itemizedlist>
<para>On the down side:</para>
<itemizedlist>
<listitem><para>we don't have anything like pkgviews. It looks like an interesting tweak,
but there's no way we can test it along what we already have.</para></listitem>
<listitem><para>we don't have anything like buildlink. This is on my todo list, once we
solved another interesting issue. We want to take control of libtool and
friends, so that we can build ports without needing to install dependencies
`for real' (libtool and pkgconfig suck, because they're hard-coded for
one specific usage pattern, which is definitely NOT OpenBSD ports building).
And the huge work that was done on buildlink will probably be needed by us
for this to work (unless my laziness forces me to have a brighter idea so
I can get away without it). The good news is that you guys already did 3
redesigns of it, so I won't have to go through all the initial mistakes
you lived through ;)</para></listitem>
<listitem><para>we have a few less ports than you do, though I dare say we now have more
or less everything that matters.</para></listitem>
</itemizedlist>
<para>And a final note: we are probably more `toolbox-oriented'. By this, I mean
that I tried really hard to give people a basic toolset that they can
combine in interesting ways, and it works, I'm surprised by some combinations
my fellow developers use. For instance, we have lots of basic introspection
commands you can combine in interesting ways, such as passing a list
of pkgpaths directly to make, and manipulating them directly.
Like, if you want to know the dependency tree of a port, you can just
do:
<screen><userinput>SUBDIR=x11/kde/libs3 make all-dir-depends</userinput></screen>
and you get a list suitable for tsort, among other things.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>And what should be adopted from the OpenBSD ports?</emphasis></title>
<blockquote>
<para>Some design ideas, probably. You are currently starting to look at
writing down build options in packages. I think that encoding that directly
in the pkgpath, and making all the tree work that way has merit.</para>
<para>I believe that you are facing some hard decisions some way down the road.
pkgsrc tries very hard to be everything for everyone. At some point, you
have to drop some tweaks so that you can concentrate on some basic stuff
correctly... some of the infrastructure has become too complicated. I
believe you should try to get rid of embedded shell-scripts in packages.
This is the exact same mistake gnu has made by the way. meta-programming
is very cool, but shipping the result is a huge mistake, because bugs get
reproduced a hundred time, and fixed in a hundred different way.
You should ship the stuff you need to reproduce this behavior instead (or
to rebuild the scripts), so that everything benefits from bug-fixes.</para>
<para>In my opinion, this is the same `mistake' Debian did: they have loads and
loads of scripts that handle all kinds of behavior. This breaks all the time,
because they cannot prepare for every contingency. But Debian has a huge
user base, and a huge pool of developers, so they can deal with this kind
of mistake, engineering-wise in `smaller' projects, this is time best
used dealing with other stuff.</para>
<para>pkgsrc should finally start killing some old stuff. Catering
to every one and every scenario is very politically correct, but at some
point, you have to break some stuff to move forward.</para>
<para>I could have some more specific advice, but that's not my place. You're free
to take pkgsrc in whichever direction you want. If you want some useful advice,
I think you should take a walk on the OpenBSD side. I don't know if your
top designers did, but I'm certain you can do the same thing I do every
day: take some good ideas from us. Get rid of the `not invented here'
syndrome.</para>
<para>I know that there is some bad blood between some NetBSD oldtimers, and some
OpenBSD people, but you can definitely get above that.</para>
<para>And stop thinking OpenBSD is just NetBSD plus a few tweaks. Especially in
the package/ports areas. I'm usually rather modest, but heck, I've
rewritten our pkgtools from scratch, and spent even more time modernizing
our infrastructure.</para>
<para>At some point, the new pkgtools was l'arlesienne, I spent over one year
poring over the design until I was satisfied, and I've sunk hundreds of
hours of development into it (plus all the tests and help I've had from my
fellow developers). I'm damn proud of the result. It's not even slow: having
a nicer language than C means I could integrate more pieces, and the resulting
commands perform faster than their C counterparts on most tasks.
Even if you don't want Perl in your base system, you can definitely
borrow ideas !</para>
</blockquote>
<para>
<emphasis><emphasis role="bold">Thank you very much for taking the time
to answer the questions!</emphasis></emphasis></para>
</formalpara>
<note>
<title>The pkgsrcCon conference</title>
<para>pkgsrcCon is the annual conference for developers and users of pkgsrc.
It takes place in Europe, since 2004. The pkgsrcCon 2007 was held in Barcelona.
As a novelty this year, the talks were recorded on video and made public. The slides
of the presentations are also available on.
<ulink url="http://www.pkgsrccon.org/">http://www.pkgsrccon.org/</ulink></para>
<para>NetBSD developer Lubomir Sedlacik talked to Will Backman about the pkgsrcCon 2007.
The podcast of the interview is
<ulink url="http://bsdtalk.blogspot.com/2007/02/bsdtalk100-netbsd-developer-lubomir.html">available
on <emphasis>bsdtalk</emphasis></ulink>
</para>
</note>
</sect3>
</sect2>
<html:hr/>
<sect2>
<sect3 id="badura">
<title>Christoph Badura: Problems and drawbacks of the pkgsrc system</title>
<formalpara>
<title><emphasis>Hi Christoph, please introduce yourself</emphasis></title>
<blockquote>
<para>I started using Unix in the mid 80s. I have been using NetBSD in
production since 0.8. I started to use pkgsrc pretty much when Alistair
Crooks created it 1997. I use NetBSD daily on the laptops that I use as
workstations.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>How do you use pkgsrc?</emphasis></title>
<blockquote>
I use pkgsrc to manage all third-party software on my NetBSD machines. I
always build binary packages and distribute them to my various machines.
I have taken to update my machine via binary packages with the help of
pkg_chk.
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Are you currently working on any pkgsrc-related projects?</emphasis></title>
<blockquote>
<para>No. I try to keep the packages up to date for which I am the maintainer
and that I still use up to date. And I try to fix bugs that I notice
along the way.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>On this years pkgsrcCon in Barcelona you did a
presentation about problems with pkgsrc for ordinary users and system
administrators
(<ulink url="http://www.pkgsrccon.org/2007/slides/bad/from-the-trenches.odp">link</ulink>).
What are the problems?</emphasis></title>
<blockquote>
<para>My idea for the talk wasn't to present a list of problems. I wanted
to give feedback to the pkgsrc developers from the point of view of an
ordinary user who doesn't have commit rights to the pkgsrc CVS tree.</para>
<para>There are a lot of things pkgsrc does right and I did mention a number of
them in my talk. There are also a number of things that pkgsrc does not do
so well and I pointed some of them out in my talk.</para>
<para>I think pkgsrc developers spend most of their time working with pkgsrc by
adding and updating packages, extending the infrastructure to make it more
portable and easier to use, and, of course, fixing bugs in the packages
and the infrastructure. I believe that that set of activities leads to
very different expectations from pkgsrc than what non-developers have.</para>
<para>As a non-developer I expect from pkgsrc that it helps me to install and
maintain all the third-party software that doesn't come with the base
operating system but which I need for the system to be useful to me.
And I want to do that with the minimum amount of time and hassle.</para>
<para>I believe that means using binary packages and good frontends to manage
the packages. In my opinion tools like aptitude and the YaST package
are a step in the right direction. I think we don't have anything equivalent
yet.</para>
<para>We also have issues with keeping the binary package collections for
the quarterly branches complete when we have to deal with security
fixes. Sometimes pre-requisite packages are removed and no replacement is
distributed.</para>
<para>We regularly move packages to a different category directory or change the
directory names for packages in pkgsrc. But we do not have tools that
deal with these re-namings automatically when one wants to update the
installed packages on the system.</para>
<para>These are the sort of issues that an non-developer will trip over.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Why is pkgsrc sometimes hard to use and what needs to be changed?</emphasis></title>
<blockquote>
<para>I don't think that pkgsrc is hard to use. Quite the contrary. In my
opinion it is very easy to use for what it does!</para>
<para>For example I was recently asked to upgrade subversion on a Linux machine
for which newer binary packages were not available. With pkgsrc it was a
snap:
<procedure>
<step><para>download and extract pkgsrc</para></step>
<step><para><userinput>cd pkgsrc/bootstrap; ./bootstrap</userinput></para></step>
<step><para><userinput>cd pkgsrc/devel/subversion-base; make install</userinput></para></step>
</procedure>
I think that is extremely easy to use!</para>
<para>Thanks to the permanent bulk builds that are going on, it is very
reliable, too. And it works not only on NetBSD but on other systems too:
Linux, Mac OS X, Solaris, IRIX, HP-UX, AIX, MS Windows, you name it. I
am not aware of any other system that does the same.</para>
<para>Of course, that assumes you want to and can build from source. When one
wants to just install precompiled binary packages pkgsrc isn't as
comfortable as other systems that are widely used today.</para>
<para>If pkgsrc wants to do better in the binary packages discipline, I think we
need more pkgsrc developers. Maintaining a working, stable and complete
set of precompiled packages is a lot of hard and time-consuming work. Much
more work than we are currently able to handle. So we need more people who
help with the security pullups to the quarterly branches, build binary
packages for all the platforms and OS versions afterwards, etc. Every
little bit helps.</para>
<para>If someone wants to join and help with the effort, if only by maintaining
2 or 3 packages, I think we should welcome them.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Your presentation confronted the pkgsrc developers with a long list
of problems. How was the feedback?</emphasis></title>
<blockquote>
<para>I think it went rather well. I didn't notice anyone falling asleep during
the talk. :-) And we had some good and lively discussions afterwards.</para>
<para>Again, the idea wasn't to confront the pkgsrc developers with a long list
of problems. But rather to show them a view of pkgsrc from a different
perspective. Hopefully some of my ideas will help them to improve the
situation.</para>
<para>But this is a volunteer effort and most developers work on what they
care about. And it is difficult to force developers on things they don't
care about. Certainly I don't expect them to suddenly start working on
my pet peeves.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Did any of the recent changes to pkgsrc improved the situation?</emphasis></title>
<blockquote>
<para>I don't know. I haven't been paying attention, though. pkgsrc is
constantly improving. So some of my issues have very likely been
addressed. I find it truly amazing how pkgsrc keeps improving and
restructuring even after ten years of existence. I don't see the same
vitality in other similar systems.</para>
<para>I believe we do not yet officially support a comfortable binary package
manager yet. But there seems to be some progress by non-developers in
pkgsrc-wip in that area.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Last question: Do you have any practical tips to share with the
pkgsrc users?</emphasis></title>
<blockquote>
<para>I recommend always building binary packages and keeping them around for a
while. Being able to downgrade to the previous version if a an updated
package has critical bugs is extremely helpful. Saving the daily
insecurity reports that log the changes to the installed packages helps
with that, too.</para>
<para>I found <ulink url="http://pkgsrc.se/pkgtools/pkg_chk">pkg_chk</ulink> by
David Brownlee to be very helpful for updating my systems with newer binary
packages and for building updated binary packages.
<userinput>make update UPDATE_TARGET=package SPECIFIC_PKGS=1</userinput>
is very useful for updating the binary packages with newer versions. I keep the
settings of the package option variables and the list of SPECIFIC_PKGS
in a file in /usr/pkgsrc and include that file from /etc/mk.conf.</para>
<para>And I always save a log file with the output of compilation, installation,
and update runs. Adding <userinput>|& tee logfile</userinput> has
become almost automatic.</para>
</blockquote>
<para>
<emphasis><emphasis role="bold">Thank you very much for taking the time
to answer the questions!</emphasis></emphasis></para>
</formalpara>
</sect3>
</sect2>
<html:hr/>
<sect2>
<sect3 id="sonnenberger">
<title><emphasis>Joerg Sonnenberger about pkgsrc on DragonFly BSD and his
pkgsrc development projects</emphasis></title>
<formalpara>
<title><emphasis>Hi Joerg, please introduce yourself.</emphasis></title>
<blockquote>
<para>I'm 23 year student of math with very broad interest in Computer
Science. I've been programming and working in various areas since high
school and joined the Open Source development when starting university.
When I don't program or study, I'm likely to either sleep, read, eat or
play chess (in that order).</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>You were the driving force behind the port of
pkgsrc to DragonFly BSD. Why did you preferred pkgsrc over
alternative systems?</emphasis></title>
<blockquote>
<para>I think there were only two options. I was fighting with keeping the
support for DragonFly in top of FreeBSD ports for quite a while. This
was getting harder and harder as DragonFly started to divert from
FreeBSD 4. It also got more work because the support for FreeBSD 4 was
already declining in FreeBSD ports at that time. At that time Todd
Willey had done enough work to get the bootstrap and some major packages
working with pkgsrc on DragonFly. When I took a more detailed look I
liked it a lot, especially as it helped me avoid a lot of redundant
work. I've started to focus on fixing things in pkgsrc, kept Thomas,
Jeremy and a few other developers busy with patches and many of the
critical items were working after a while. I don't regret that choice
and I am faithful that I choose the technically better system. The
biggest complaint at that time about pkgsrc (compared to FreeBSD ports)
was the lack of portupgrade or equivalent.
<ulink url="http://pkgsrc.se/wip/pkgmanager">pkgmanager</ulink> in wip
and <ulink url="http://pkgsrc.se/pkgtools/pkg_rolling-replace">pkg_rolling-replace</ulink>
somewhat provide that now, at least enough to stop
many of the louder voices.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>In 2006 you said that porting pkgsrc to DragonFly was like
<ulink url="http://www.pkgsrccon.org/2006/slides/pkgsrc-on-df/index.html">"Fighting the Windmills"</ulink>.
What were the problems? Was pkgsrc not portable enough?</emphasis></title>
<blockquote>
<para>A mix of both. The biggest problem with pkgsrc itself is (and was) that
DragonFly does a number of things different, e.g.
<emphasis> pthread linkage</emphasis>. Hunting down all those instances
took a while and they still pop up occasionally. There are also a huge number of
portability issues in programs, like the infamous *bsd* pattern in configure scripts.
If you haven't read the slides from the presentation, there are quite a few
number of funny problems I meanted back then.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>But it seems that pkgsrc works quite well on
DragonFly these days.</emphasis></title>
<blockquote>
<para>Yeah, I'm trying to keep the breakage down to a minimal level. There was
a time when DragonFly was between NetBSD 3.1/i386 and NetBSD 4/i386 in
terms of broken packages, which is quite nice. I think the overall state
is quite nice, setting the right PKG_PATH and using <emphasis>pkg_add</emphasis>
is enough work to get a full scaled system in a short time.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>How did the pkgsrc system benefit from the
porting efforts?</emphasis></title>
<blockquote>
<para>DragonFly helped to sort out imake issues, a number of missing
libtool overwrites, GCC 3.4 issues, pthread linkage etc. It is also very
important to make the portability aspect of pkgsrc a reality, not just a
lip service.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>You are very active in developing for pkgsrc. Please tell us
about your pkgsrc projects so far</emphasis></title>
<blockquote>
<para>The single biggest item so far was the addition of
<emphasis>modular Xorg</emphasis> support
without breaking the tree for anyone else. The work in that area started
early this year and until the official announcement mails no-one would
have known without reading the CVS commit mails. That is quite different
from how it was handled e.g. in FreeBSD ports with a long ports freeze
and a lot of pushing to get everyone to switch at once.</para>
<para>It is hard to say which project I've been working on as so much is going
on. At least one other major change I would like to get credit for. We
had a system to use different Apache and PHP versions for a long time.
Until recently, it was impossible for a binary package user to freely
select his or her choice, because only one set was built and even worse,
the name was identical. Inspired by the way our Python modules deal with
this problem, almost all Apache and PHP modules were changed to be
prefixed with the version and have explicit dependencies as well. In
combination with pbulk this makes it possible to choose Apache 1.3 and
PHP 4, as well as Apache 2.2 and PHP 5 -- without building from source.</para>
<para>In the more recent past, the
<ulink url="http://netbsd-soc.sourceforge.net/projects/xpkgsrc/">Summer of Code</ulink>
has kept me busy. Both the new bulk build framework and the cross
compiling work are in the tree now and ready for use. They still need
polishing, but what doesn't need
that? :-)</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>What are you currently working on?</emphasis></title>
<blockquote>
<para>I'm actively working on re-factoring the pkg_install code with the goal
of making pkg_add and pkg_info use libarchive like pkg_create already
does. This is non-trivial work as the code has a huge number of side
effects and it is very easy to break things one way or the other. Some
of the regressions are just exposed bugs that were never that critical
before, others are misunderstandings of how it was thought to work.
The benefit for the user is that pkg_add will be much faster. If the
prototyping Tim Kientzle did in FreeBSD once can be used as foundation,
a factor of 2 in speed is possible and more. This is also very useful
for bulk builds as some parts are currently IO bound like building many
of the smaller packages.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>And the future plans?</emphasis></title>
http://www.pkgsrccon.org/2007/
<blockquote>
<para>More pkg_install work. We had some long discussions in
<ulink url="http://www.pkgsrccon.org/2007/">Barcelona</ulink> and I
want to adopt some of the ideas from pkg_jam. A better pkgdb is one of
the biggest problems for handling updates more gracefully etc. I try to
not plan ahead too much though.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>One of your current projects is the
"Cross-compiling of modular Xorg". What's the goal, the state and what
will this mean for the NetBSD users. Will there be no more xsrc as part
of the base system?</emphasis></title>
<blockquote>
<para>The goal of the cross-compiling support was to create a migration path
from xsrc to modern Xorg. It was a hotly discussed topic and only two
options were really sensible. The first was to extend the reach-over
framework to deal with modular Xorg as well. This was started, but never
finished. It would also create some maintenance issues as pkgsrc has to
maintain a second set anyway. The second option was to always use pkgsrc
and for that it has to allow cross-compilation. That was the goal for
this part of the <emphasis>Summer of Code</emphasis> project. The code is
integrated into pkgsrc and the tests were quite successful. Remaining issues are
documentation and integration with either pbulk or build.sh. For xsrc the server
component for all the non-standard hardware on Sparc and older platforms has
to be packaged so that we can build them from pkgsrc. Once that is done,
xsrc can be retired.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>A longtime goal was to make pkgsrc a cross-compilable system like
NetBSD. Is this goal now abandoned?</emphasis></title>
<blockquote>
<para>In general, yes. There are a number of components that simply
can't be cross-compiled in a sane manner. Anything with a large complex
build system for example. I'm not sure if it is possible to cross-compile Perl
or Python for example. Even more important would be the upstream
interest in such patches -- this is definitely something we don't want
to maintain in pkgsrc. Having said that, we can support
cross-compilation for many packages, both smaller and larger ones.
Modular Xorg is one specific set where NetBSD has an interest in,
others are possible. If there's demand, it can be extended to
cover more.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>For the ordinary pkgsrc user: What were the most
important changes in the last two years?</emphasis></title>
<blockquote>
<para>The addition of modular Xorg and the on-going efforts to improve binary
packages and build consistency based on the check framework. The former
is just a requirement for modern hardware and the latter makes the risks
of using current quite a bit lower over all.</para>
<para>The DESTDIR support could have quite some impact, but it needs some more
work to be supported on a broader scale and be advertised as such.</para>
</blockquote>
<para>
<emphasis><emphasis role="bold">Thank you very much for taking the time
to answer the questions!</emphasis></emphasis>
</para>
</formalpara>
</sect3>
</sect2>
<html:hr/>
<sect2>
<sect3 id="lowden">
<title>James K. Lowden: The pkgjam project</title>
<formalpara>
<title><emphasis>Hi James, please introduce yourself.</emphasis></title>
<blockquote>
<para>I'm old enough that the computer I use nowadays is better in all
dimensions by 4 orders of magnitude, far superior to the clay tablets we
had in school. I make my living as a quant on Wall Street. Two beautiful
daughters, each smarter and more accomplished than the other. NetBSD user
since 1999, maintainer for the <firstterm>FreeTDS</firstterm> project, and
insufferable optimist.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Together with pkgsrc developer Johnny C. Lam you started
the pkgjam project (<ulink url="http://www.pkgjam.org/">project homepage</ulink>),
aimed as the successor of the pkgsrc system. Why is a re-design of pkgsrc
necessary?</emphasis></title>
<blockquote>
<para>The weaknesses of pkgsrc -- and I speak as a fan and user -- are well
known. It's hard if not impossible to replace installed packages without
disabling your system for a while, sometimes for days, because installed
packages are normally deinstalled prior to even building their
replacements. Also, it's difficult to interrogate the system about
installed (or available) packages beyond what's provided by
<firstterm>pkg_info(1)</firstterm>.</para>
<para>The basic pkgsrc architecture was designed when 8 MB of RAM was a lot.
Nowadays pkgsrc has over 7000 packages and it's certainly not uncommon to
have hundreds of packages installed. My main machine -- clearly the
prototypical case, obviously -- has 356 installed packages. That's a lot
of interdependency. Our systems are more complex than they used to be,
and they're more powerful, but the basic tools of pkgsrc -- make and
shell, and the Berkeley database -- are unchanged.</para>
<para>No one pretends that <firstterm>make(1)</firstterm> is the be-all and
end-all of complexity management tools, but it lives at the center of pkgsrc.
(At the pkgsrcCon in Barcelona last April, it was estimated that 50% of the time
building packages was spent not in compiling and linking, but in
processing Makefiles.) To take make(1) out of its starring role in pkgsrc
is to redesign it, so that's where we headed. </para>
<para>pkgsrc is no developer's picnic, either. Anyone who works on the project
will tell you it requires a lot of knowledge about the tools and about the
project's conventions. Johnny and I both believe that a lot of pkgsrc's
internal complexity results from using make(1) for things it was never
meant for. We hope pkgjam will be easier for developers to grok and join.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>What are the main concepts and technologies that pkgjam is based on?</emphasis></title>
<blockquote>
<para>We can reduce package complexity and manage the remaining complexity
better. Let's apply some of the lessons learned from 10 years of pkgsrc
and some of the tools now available to make the system faster, more
transparent, and more robust.</para>
<para>To that end, pkgjam has three main features:</para>
<orderedlist>
<listitem><para>
<firstterm>Dependency Independence</firstterm>: If packages B and C both rely on
package A, and you want to upgrade package B, which requires upgrading package A,
need you upgrade package C, too?
Logically, no. Technically, no, too, as long as you're prepared to
have two versions of package A installed (one for B and one for C). We reduce one
kind of complexity, interdependence, by introducing another, package
proliferation. But the system becomes more modular, more robust.
Especially if you can install B 2.2 while B 2.1 remains installed and
working.</para></listitem>
<listitem><para>
<firstterm>Relational Database for package metadata</firstterm>: This is how we attack
the remaining complexity. All package metadata -- build-time options and
dependencies, licences, everything you need to know to build a package --
is stored in the database. So is information about packages you installed
and the options you built them with (and their dependencies, of course).
The database also has all your preferred options, settings that influence
how packages are built. pkgjam uses the database to track dependencies
and to generate build plans. Users and developers can query the database
with <firstterm>SQL</firstterm> to answer questions not supported
by <firstterm>pkg_info(1)</firstterm>.</para></listitem>
<listitem><para>
<firstterm>New Build Tool</firstterm>: The new build tool is what's
invoked by the user to build and install packages. It relies on the database
to decide what to do, and relies on make(1) to do what it does best:
build and installpackages. </para></listitem>
</orderedlist>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>One feature of pkgjam is the "Dependency Independence":
every package is installed in its own directory. Is this concept inspired by the
"Application Directories" from RISC OS or the "Bundle"
concept from NEXTSTEP?</emphasis></title>
<blockquote>
<para>I have no experience with <firstterm>NEXTSTEP</firstterm> or
<firstterm>RISC OS</firstterm> and didn't even pretend to
try to research prior art. It was clear to both of us from the start that
what I call "the one true tree" made using pkgsrc harder than it had to
be. Johnny has been a pkgsrc developer for years; I approached the
package metadata problem as a user. Johnny also had experience from
working on <firstterm>pkgviews</firstterm>. You could say, informally,
that he's the problem domain expert and I'm the data analyst.</para>
<para>When we started to sketch out our ideas to each other, we looked a little
into how <firstterm>OS X</firstterm> does things. OS X has a slightly different
notion of how an application finds its shared libraries: approximately, it permits the
RPATH to have a relative path. That lets an OS X user pick up his application and
drop it anywhere. The whole subtree moves, and the application looks only down its
tree for its shared objects.</para>
<para>NetBSD and, as far as I know, all the free operating systems we see pkgjam
supporting one day, use <firstterm>ELF binaries</firstterm> and expect
absolute <firstterm>RPATHs</firstterm>. On such
systems, you can move the executable, but not its shared objects. I'm
told that this is a kernel property and not particularly hard to change.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>And what about the handling of shared libraries, is every
package installed with its own libs?</emphasis></title>
<blockquote>
<para>Basically, yes. A shared library is first installed in its canonical
directory. Dependent applications do not refer to it there, however.
Instead, a link to the shared object is created in the application's tree.
The application's RPATH points there.</para>
<para>We use a hard link to the shared object because we want to be able to
delete the original and replace it with an updated (or just differently
configured) version. We want to be able to do
<emphasis role="bold">anything</emphasis> with a shared
library package without disturbing its dependents.</para>
<para>In practice, there are some constraints. Some libraries, for example,
refer to a database whose location is relative to the install directory,
or otherwise insist on using a relative path to find some resource. Such
packages can't be freely deleted and replaced. But that's a feature of
the package, not of the package management system.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>How will the use of pkgjam be different from pkgsrc?</emphasis></title>
<blockquote>
<para>Johnny and I both like the simplicity of pkgsrc's
<userinput>cd <pkg> && make && make
install</userinput>. There'll be more separation between building and
installing. The method of expressing system-wide preferences and
package-specific options will be very different, because they'll be kept
in the database instead of make variables.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Which new and better solutions will pkgjam provide for the ordinary users
and system administrators?</emphasis></title>
<blockquote>
<para>Today, to install package X version N, you have to first deinstall N-1.
pkgjam will let you build version N while using N-1. It will let you
install N without erasing N-1. Reverting is just a matter of removing the
links to N and recreating the links to N-1.</para>
<para>Dependencies are all in the database, making it possible to view what
would be affected by an action before it's taken. I don't know of any
package system that does anything like that in a way that's comprehensible
to a human being.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>You presented pkgjam on this years pkgsrcCon in Barcelona. How was the
feedback from the developers community?</emphasis></title>
<blockquote>
<para>Some mild skepticism, but generally great enthusiasm. So much so that I
changed direction, focusing more on making pkgjam ready for others to
join and use, and less on automatically generating "easy" packages.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>About the current state of pkgjam: is it ready to use?</emphasis></title>
<blockquote>
<para>It's alpha software. If you're interested in developing a packaging
system and you like the pkgjam approach, then we have a skeleton you can
use that we're building on. I said in Barcelona we'd have something ready
by July, and we do have something. We took a break over the summer. Stay
tuned.</para>
</blockquote>
</formalpara>
<formalpara>
<title><emphasis>Please tell us about your roadmap and about
future perspectives.</emphasis></title>
<blockquote>
<para>We're using an iterative approach: build something, use it, tear it down,
do it again. We want the interface to the database to be well defined for
the build tool, such that there could be other build tools (say, a GUI or a
web-based one). We want the packager's job to be as declarative as
possible. The packager shouldn't have to become a database expert.</para>
<para>As our package list grows, the database and its interface will stabilize,
and people will, unless we build the next Edsel, start using pkgjam. That
will create a need for canned reports, something like pkg_info(1). With
luck, early adopters will have SQL knowledge and will be able to
contribute and collaborate on new ways to use the database.</para>
<para>pkgjam will be "done" when every useful pkgsrc package is also a pkgjam
package. To get there, we'll need packagers and users. To attract them,
we need to make packaging easy and package management better. Which is
fine, because that was our goal in the first place. Because, you know,
I'm not just a pkgjam developer. I'm also a client.</para>
</blockquote>
<para>
<emphasis><emphasis role="bold">Thank you very much for taking the time
to answer the questions!</emphasis></emphasis>
</para>
</formalpara>
<note>
<title>The concepts of <emphasis>Application Directories</emphasis></title>
<para>Self-containing application directories first appeared on the NEXTSTEP and RISC OS operating
systems: All files needed by an application are grouped together into one directory. Application directories
are known as
<emphasis><ulink url="http://en.wikipedia.org/wiki/Bundle_(NEXTSTEP)">Application Bundles</ulink></emphasis>
under NEXTSTEP, GNUstep, Mac OS 9 and Mac OS X.</para>
<para>Various opensource projects use similar concepts: The
<ulink url="http://rox.sourceforge.net/desktop/about_rox">ROX Desktop</ulink> adopted the Application
Directories from RISC OS. The <ulink url="http://klik.atekon.de/">klik</ulink> software install system for
Linux and the <ulink url="http://www.pcbsd.org/content/view/20/26/">PBI system</ulink> for PC-BSD are
independent implementations. <ulink url="http://nix.cs.uu.nl/">Nix</ulink> is an advanced,
"purely functional package manager" for Unix systems, each version of a package is stored in its own directory. On
<ulink url="http://www.gobolinux.org">GoboLinux</ulink>, a Linux distribution with an alternative directory structure,
each program gets its own directory tree.</para>
</note>
</sect3>
</sect2>
<!-- TEMPLATE:
<formalpara>
<title><emphasis></emphasis></title>
<blockquote>
</blockquote>
</formalpara>
-->
</sect1>
<html:hr/>
<sect1 id="contact">
<para>Feedback, corrections and additions are always welcome, please contact me via
<ulink url="mailto:mark.weinem@alumni.uni-due.de">email</ulink>.</para>
<para>Many thanks to all participants!</para>
<para>
<emphasis role="bold">Please note that the expressed opinions and
the selection of topics do not represent the official views and
directions of the pkgsrc project.</emphasis>
</para>
<para>- Mark Weinem</para>
</sect1>
</webpage>