Packages Not [was Re: ambivalence about fortran (was Re: numpy & non-Apple gcc?)]

[Jordan K. Hubbard]

On Sep 21, 2010, at 1:29 AM, Anders F Björklund wrote:

> This is what RPM does already... rpmbuild(8) builds, rpm(8) installs.
> 
> Building has BuildRequires, Installing has Requires. Otherwise both use .rpm packages (for digests, signatures, etc). There's even separate types of binary packages. Some packages have really long and complex needs for building from source, but installing the binary does not. So having "foo" and "foo-devel" helps cut down the number of dependencies, when not always wanting to install the developer files too (like done with ports). Libraries can also be split off from programs, same reason.

Yep, RPM already has this whole ecosystem covered - SRPMs are their "ports" and the RPMs their packages.  However, I think we can also all agree that the "spec" format for declaring all of this has always been pretty ugly (I don't know what Jeff has in store for RPM's future, but right now we have what we have) and we don't necessarily want to target it without a good reason.  Arguably, we also have a really good build system already, so all of those portions of RPM would be redundant extra code to carry around, in a certain sense, and what we really need are just the package delivery bits so RPM is probably overkill.

>> So, as complicated as everyone seems to like to make this every time the subject comes up, the actual task to be done is fairly simple.   MacPorts needs to work essentially like this:
>> 
>> port install foo
>> entails:
>> 	fetch foo
>> 	extract foo
>> 	configure / build foo
>> 	"install" foo to destroot
>> 	make package from destroot (say, just for discussion's sake, that package goes into /tmp in this scenario)
>> 	cleanup destroot and basically "finish" the macports phase
>> 
>> 	package-install /tmp/foo.xpkg
> 
> Where the only difference to what it does now is that the final install step is separate from the others, so that you can start with just "pkg(1)" and "*.xpkg" and not have to do all the rest ?

Correct!   More to the point, you should also be able to do something like:   pkg -R -add someCoolPackage.xpkg in order to connect to a remote HTTP / FTP server somewhere and just grab the latest version of someCoolPackage for your relevant OS / architecture, installing any dependencies that someCoolPackage might have automatically.

This isn't just a weird idea - *BSD's pkg_add command currently supports this with the -r flag, e.g.:

freebsd8# pkg_add -r bash
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.1-release/Latest/bash.tbz... Done.
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.1-release/All/libiconv-1.13.1_1.tbz... Done.
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.1-release/All/gettext-0.18_1.tbz... Done.

I can install FreeBSD 8.1 for AMD64/SPARC/PowerPC on the relevant hardware, for example, and then as the very first thing on the new box "pkg_add -r bash" and it just works, automatically appending the architecture and FreeBSD version # to a built-in search path which is used to search around at ftp://ftp.freebsd.org/pub/FreeBSD/packages/ until the most applicable bash package can be found (I think it looks in about 5 places, in decreasing order of specific relevance).  I don't have to know or care which OS revision or architecture I'm on if the nice package makers over at FreeBSD.org have taken my needs into account with their package building fleet, and it's all very convenient and easy to use - my internet connection is the only bottleneck in quickly populating my system with the extra software I need.  

> Archives are different in that they only allow you to shortcut the "destroot" phase, but do need all of the others. Packages would need to have all runtime information available, without peeking in the port. I think you are missing the "activate" step above, but then again I don't think that step is very useful.

The activate step was a wrinkle I kind of deliberately glossed over. :-)   We'd have to decide, in a package producing world, just what to do about the different installation types, direct and image, and perhaps make some hard decisions about perhaps just supporting one of them.  I can certainly see packages being somewhat polymorphic in how they deal with direct/image, doing different things at install time (I even have a pretty fair idea as to how to do it), but it still begs the question as to whether the additional complexity would really be worth it or if we couldn't just say "to hell with it, packages are always direct mode installs!".  Food for thought.

> I'm not sure why the runtime has to be in the packages themselves, it seems less redundant to have a small "installer" program installation contain that ? But either way, it would have been a "MacPorts Developer" that would offer to build things from ports and a "MacPorts User" that would simply install pre-made archives. And for that to work better, archives would need some improvements (extra data).

I also don't know how far the pkg tool and xpkg format have come, and I would be the last person to argue for some crufty legacy system in favor of something truly new and shiny, but if we're a ways from seeing home-grown tools and people are sick and tired of hearing me talk about packaging all the time, I would be more than willing to port the FreeBSD package tools over since I know how they work and what their OS dependencies are, then enhance archives to the point to where the +CONTENTS file is actually meaningful and relevant to the task of installing packages.  At that point, archives really would be packages since you could scrape them all together and put them on an FTP / HTTP site for use by pkg_add -r, as demonstrated earlier.  I would probably have to contrive a new Portfile variable for expressing some of the extra metadata which can't be easily contrived from inspecting ${DESTROOT}, and for some period of time this would "parallel" the post-install/post-activate procedures in some of the ports until we could retire one in favor of the other, but that's also the easiest transition to make which doesn't break anything.

> It just seems like a lot of work, making a new package manager and reinventing digests and signatures and embedded scripts and storage formats ?

Not if we start with something existing.  See above.

> Could just be that the FreeBSD OS is easier to handle than the Darwin OS (talking about the "pure" variant!), or that Ruby is nicer than Tcl for writing code other than the build recipes. Perhaps it is the available releases with binary packages, or the stricter porting. It is definitely _not_ the KDE, as I would normally be using Xfce with either.

Well, the increasing popularity of HomeBrew probably says something about the popularity of Ruby vs Tcl, but I would also maintain that most folks writing Portfiles don't even really know they're using Tcl since the language looks so much like standard english ("frob some arguments") in its simpler form.  I don't think Tcl is the biggest barrier to entry for most users, simply the lack of pre-built software which mirrors their experience on other platforms (including the iPhone :) ).

> And I didn't think MacPorts cared about making binary packages…

I think MacPorts "cares" for some value of care, it's just waiting for someone else to do the work.  I'm even volunteering, since I'm tired of always just ranting about it and then wandering off again. ;-)   Of course, there are always dissenting opinions:

On Sep 21, 2010, at 5:08 AM, Takeshi Enomoto wrote:

> Those who prefer binaries could use Fink or Homebrew.
> To me MacPorts is an aid to build open source software.
> It is better than free format instructions and binary packages that
> could be inconsistent with others.
> Even with consistent binaries it would not be easy to provide
> every possible combination of variants.

Which to me suggests kind of an admission of defeat.  "We can't provide packages, so why not go use Fink or Homebrew?"   I think that really under-sells what MacPorts is capable of, and one reason I *don't* just use Fink or Homebrew has a lot to do with the quality of the build system and my faith in its ability to create software which has been compiled in an appropriate "jail" such that I know it's going to work.  I also have a lot more faith in MacPorts' small army of volunteers keeping the build recipes up to date and otherwise making sure it works on a reasonable number of Darwin platforms, I just also want to offer that magic "pkg_add -r" functionality such that building stuff is more of a step of last resort.

As to the variants, I would agree that they're harder to create packages for since you would have to iterate through every possible package/variant combination to truly generate a package collection which anyone could use, but that's perhaps where the idea of an "on demand package builder" makes sense, since you then only have to build and cache stuff users are actually asking for, and this would also allow you to keep stats on the most popular packages (globally) such that people who don't actually know what variants  to ask for can simply look at what other people are using and pick the most popular variant configuration.

On Sep 21, 2010, at 10:00 AM, Landon J Fuller wrote:

> I agree with just about everything Jordan has had to say, but I also agree that it would be a shame to put the breaks on an emerging pragmatic solution. As long as MacPorts doesn't *require* a development environment for end-users, then using signed archives -- as long as they include the same metadata a port install would -- seems like a good first step.

Thanks, Landon.  I think that first step is actually fairly straight-forward, and unless Anders would like to pursue pkg(1) + xpkg format, which I would also enthusiastically support, I'll be happy to do the work.  All this management is making my brain turn to sludge and I could use the distraction. :-)

- Jordan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macosforge.org/pipermail/macports-dev/attachments/20100922/4a90974a/attachment-0001.html>