Packages Not [was Re: ambivalence about fortran (was Re: numpy & non-Apple gcc?)]

[Anders F Björklund]

Jordan K. Hubbard wrote:

>> This is what RPM does already... rpmbuild(8) builds, rpm(8) installs.
>>
>> Building has BuildRequires, Installing has Requires. Otherwise  
>> both use .rpm packages (for digests, signatures, etc). There's  
>> even separate types of binary packages. Some packages have really  
>> long and complex needs for building from source, but installing  
>> the binary does not. So having "foo" and "foo-devel" helps cut  
>> down the number of dependencies, when not always wanting to  
>> install the developer files too (like done with ports). Libraries  
>> can also be split off from programs, same reason.
>
> Yep, RPM already has this whole ecosystem covered - SRPMs are their  
> "ports" and the RPMs their packages.  However, I think we can also  
> all agree that the "spec" format for declaring all of this has  
> always been pretty ugly (I don't know what Jeff has in store for  
> RPM's future, but right now we have what we have) and we don't  
> necessarily want to target it without a good reason.  Arguably, we  
> also have a really good build system already, so all of those  
> portions of RPM would be redundant extra code to carry around, in a  
> certain sense, and what we really need are just the package  
> delivery bits so RPM is probably overkill.

I don't think the yaml-based .spec format is _that_ much uglier than  
the text-based .port, but that is all handled behind the scenes  
anyway since the MacPorts build system stays the same. This was more  
about "the package delivery bits". Splitting packages into  
subpackages is optional with RPM, not required. OpenPKG doesn't do  
it, for instance (they have no *-devel).

But you know where RPM is, if wanted. The previous versions are at  
http://rpm4darwin.sourceforge.net/ (or just use "port install rpm")  
or http://rpm5.org/files/rpm/rpm-5.2/BINARY/RPM5-20090707.dmg (port  
"rpm52"), or build the new stuff for Snow Leopard from source on  
http://rpm5.org/sources.php for all the new features that Jeff  
mentioned... If wanting more than archives.

>> Where the only difference to what it does now is that the final  
>> install step is separate from the others, so that you can start  
>> with just "pkg(1)" and "*.xpkg" and not have to do all the rest ?
>
> Correct!   More to the point, you should also be able to do  
> something like:   pkg -R -add someCoolPackage.xpkg in order to  
> connect to a remote HTTP / FTP server somewhere and just grab the  
> latest version of someCoolPackage for your relevant OS /  
> architecture, installing any dependencies that someCoolPackage  
> might have automatically.
>
> This isn't just a weird idea - *BSD's pkg_add command currently  
> supports this with the -r flag, e.g.:
>
> freebsd8# pkg_add -r bash
> Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.1- 
> release/Latest/bash.tbz... Done.
> Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.1- 
> release/All/libiconv-1.13.1_1.tbz... Done.
> Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.1- 
> release/All/gettext-0.18_1.tbz... Done.
>
> I can install FreeBSD 8.1 for AMD64/SPARC/PowerPC on the relevant  
> hardware, for example, and then as the very first thing on the new  
> box "pkg_add -r bash" and it just works, automatically appending  
> the architecture and FreeBSD version # to a built-in search path  
> which is used to search around at ftp://ftp.freebsd.org/pub/FreeBSD/ 
> packages/ until the most applicable bash package can be found (I  
> think it looks in about 5 places, in decreasing order of specific  
> relevance).  I don't have to know or care which OS revision or  
> architecture I'm on if the nice package makers over at FreeBSD.org  
> have taken my needs into account with their package building fleet,  
> and it's all very convenient and easy to use - my internet  
> connection is the only bottleneck in quickly populating my system  
> with the extra software I need.

I don't care very much for pkg_add(1) directly, as it's a bit too  
lowlevel for me. I prefer using portinstall(1) from pkgtools, with  
the -PP (--use-packages-only) option if I'm looking for binaries...

But yeah it is nice, the main missing features are the package size  
or the package contents. But that is easy enough to add with  
additional metadata beyond the INDEX (such as FILE_LIST, or MANIFEST)

>> Archives are different in that they only allow you to shortcut the  
>> "destroot" phase, but do need all of the others. Packages would  
>> need to have all runtime information available, without peeking in  
>> the port. I think you are missing the "activate" step above, but  
>> then again I don't think that step is very useful.
>
> The activate step was a wrinkle I kind of deliberately glossed  
> over. :-)   We'd have to decide, in a package producing world, just  
> what to do about the different installation types, direct and  
> image, and perhaps make some hard decisions about perhaps just  
> supporting one of them.  I can certainly see packages being  
> somewhat polymorphic in how they deal with direct/image, doing  
> different things at install time (I even have a pretty fair idea as  
> to how to do it), but it still begs the question as to whether the  
> additional complexity would really be worth it or if we couldn't  
> just say "to hell with it, packages are always direct mode  
> installs!".  Food for thought.

That is what the current package targets (pkg, rpm) do, as they don't  
have any images. Image mode is only useful if you can have multiple  
versions active at the same time, and since you can't they're just  
sitting there wasting space. At least the packages are archived and  
compressed, but still there without rebuild if you need to reinstall.

>> I'm not sure why the runtime has to be in the packages themselves,  
>> it seems less redundant to have a small "installer" program  
>> installation contain that ? But either way, it would have been a  
>> "MacPorts Developer" that would offer to build things from ports  
>> and a "MacPorts User" that would simply install pre-made archives.  
>> And for that to work better, archives would need some improvements  
>> (extra data).
>
> I also don't know how far the pkg tool and xpkg format have come,  
> and I would be the last person to argue for some crufty legacy  
> system in favor of something truly new and shiny, but if we're a  
> ways from seeing home-grown tools and people are sick and tired of  
> hearing me talk about packaging all the time, I would be more than  
> willing to port the FreeBSD package tools over since I know how  
> they work and what their OS dependencies are, then enhance archives  
> to the point to where the +CONTENTS file is actually meaningful and  
> relevant to the task of installing packages.  At that point,  
> archives really would be packages since you could scrape them all  
> together and put them on an FTP / HTTP site for use by pkg_add -r,  
> as demonstrated earlier.  I would probably have to contrive a new  
> Portfile variable for expressing some of the extra metadata which  
> can't be easily contrived from inspecting ${DESTROOT}, and for some  
> period of time this would "parallel" the post-install/post-activate  
> procedures in some of the ports until we could retire one in favor  
> of the other, but that's also the easiest transition to make which  
> doesn't break anything.

The pkg(1) tool was just a copy/paste of port(1) tool, with lots of  
sections ripped out and a shortcut to handle file arguments (rather  
than port names). This was so you could say "pkg install  
foo-1.2_3.tgz" or "pkg install http://example.com/foo-1.2_3.tgz",  
rather than "pkg install foo" and use the pkgindex. But it didn't  
make it as far as a separate install or anything.

The xpkg format "version 0.2" was added in MacPorts 1.8.0, it would  
need similar improvements to +CONTENTS. Porting the needed fields  
over from FreeBSD sounds like a good transition, but pkg would  
probably still need to embed Tcl for the more complex package  
scripts. And yes, it would still all be "archives". The main  
difference with xpkg was the XML...

>> It just seems like a lot of work, making a new package manager and  
>> reinventing digests and signatures and embedded scripts and  
>> storage formats ?
>
> Not if we start with something existing.  See above.

Archives didn't have digests or signatures, but could use something  
simplistic like "CHECKSUMS.md5.asc" or "Release.gpg". That is, just  
have GNU Coreutils digest all the package containers into one index  
and then sign that file with GnuPG, like Slackware or Debian does it.

Or use .rmd160 and openssl, for each archive file ? (like http:// 
trac.macports.org/browser/trunk/base/src/package1.0/ 
portarchivefetch.tcl) Depends on what new fields you end up with for  
the pkgdata and pkgindex, I suppose...

> Well, the increasing popularity of HomeBrew probably says something  
> about the popularity of Ruby vs Tcl, but I would also maintain that  
> most folks writing Portfiles don't even really know they're using  
> Tcl since the language looks so much like standard english ("frob  
> some arguments") in its simpler form.  I don't think Tcl is the  
> biggest barrier to entry for most users, simply the lack of pre- 
> built software which mirrors their experience on other platforms  
> (including the iPhone :) ).

I prefer my recipes to be markup rather than scripts, but I'm OK with  
either scripting language (Sh, Tcl, Ruby) for the build scripts.

It's not needed for "base" to be in Tcl too, just because the  
"dports" tree is, but I think that is also being fixed now (with  
Objective-C).

>> And I didn't think MacPorts cared about making binary packages…
>
> I think MacPorts "cares" for some value of care, it's just waiting  
> for someone else to do the work.  I'm even volunteering, since I'm  
> tired of always just ranting about it and then wandering off  
> again. ;-)   Of course, there are always dissenting opinions:

Then again, why *should* MacPorts care about making binary packages  
that can be installed without MacPorts ? That was the main argument  
for going with archives, as they were "internal".

But actually building binaries, or fixing licensing issues such as  
openssl, was much more important than discussing storage formats or  
logging formats or bikeshed colors. Thus: archives.


> On Sep 21, 2010, at 5:08 AM, Takeshi Enomoto wrote:
>
>> Those who prefer binaries could use Fink or Homebrew.
>> To me MacPorts is an aid to build open source software.
>> It is better than free format instructions and binary packages that
>> could be inconsistent with others.
>> Even with consistent binaries it would not be easy to provide
>> every possible combination of variants.
>
> Which to me suggests kind of an admission of defeat.  "We can't  
> provide packages, so why not go use Fink or Homebrew?"   I think  
> that really under-sells what MacPorts is capable of, and one reason  
> I *don't* just use Fink or Homebrew has a lot to do with the  
> quality of the build system and my faith in its ability to create  
> software which has been compiled in an appropriate "jail" such that  
> I know it's going to work.  I also have a lot more faith in  
> MacPorts' small army of volunteers keeping the build recipes up to  
> date and otherwise making sure it works on a reasonable number of  
> Darwin platforms, I just also want to offer that magic "pkg_add -r"  
> functionality such that building stuff is more of a step of last  
> resort.

So the first step would be finishing the MPAB project ?

> As to the variants, I would agree that they're harder to create  
> packages for since you would have to iterate through every possible  
> package/variant combination to truly generate a package collection  
> which anyone could use, but that's perhaps where the idea of an "on  
> demand package builder" makes sense, since you then only have to  
> build and cache stuff users are actually asking for, and this would  
> also allow you to keep stats on the most popular packages  
> (globally) such that people who don't actually know what variants   
> to ask for can simply look at what other people are using and pick  
> the most popular variant configuration.

Just building the default variants sounds like a start.


> On Sep 21, 2010, at 10:00 AM, Landon J Fuller wrote:
>
>> I agree with just about everything Jordan has had to say, but I  
>> also agree that it would be a shame to put the breaks on an  
>> emerging pragmatic solution. As long as MacPorts doesn't *require*  
>> a development environment for end-users, then using signed  
>> archives -- as long as they include the same metadata a port  
>> install would -- seems like a good first step.
>
> Thanks, Landon.  I think that first step is actually fairly  
> straight-forward, and unless Anders would like to pursue pkg(1) +  
> xpkg format, which I would also enthusiastically support, I'll be  
> happy to do the work.  All this management is making my brain turn  
> to sludge and I could use the distraction. :-)

Well, MacPorts *does* require Xcode now. (http://trac.macports.org/ 
changeset/68593) Even if all you want to do is install archives... It  
also requires a full ports tree, and even does a "sync" for you  
automagically. When using packages you would probably have a  
(similar) "pkgindex" rather than the "portindex" (+Portfiles), once  
archives have all metadata. The "selfupdate" could simply link to the  
download webpage.

I don't think pursuing your xpkg format is worthwhile, IMHO it (.xar  
and .xml) doesn't offer anything over the usual (.tar and .txt)  
besides worse compression ratios and more complex parsing. Adding pkg 
(1) as a complement to port(1) is still doable, but one can just use  
"port -b" meanwhile ? Of course, the first thing required for that to  
work would be to enable "archivemode" which is currently disabled by  
default.

--anders