Archives and Packages (was Re: Universal and binary builds)

[Anders F Björklund]

Jordan K. Hubbard wrote:

>> This would still require someone to choose/write the "pkg" program,
>> and then make all invocations of "install" go through this layer...
>
> Yep!  That's the basic idea.

Unfortunately choosing an existing program (such as rpm) means that  
it isn't optimally built for the task, and building a whole new  
program (such as xpkg) complicates the task enormously.

So at this point it has only built packages missing a few things  
(such as the variants or images) for an external program, or built  
complete packages for an internal program that is missing a few  
things (such as an implementation). And neither was very  
"popular" (slight understatement) with users that want to build from  
source anyway, and were not using packages. At least that was where  
it left off, as I remember it.

>> And both using the RPM package manager and writing a new one was  
>> tried
>> (with "dp_light" and "xpkg"), neither solution has been very  
>> popular ?
>
> I don't know if "popular" would be the word I would use.  Neither  
> solution has ever actually been *completed* would be a more  
> accurate statement, so the resulting popularity of said solution is  
> impossible to gauge.  The MacPorts project has always had a rather  
> lackluster attitude when it comes to the challenge of packaging up  
> and delivering software to end-users, but the fact remains that  
> most end-users just want to say "I want this, that and that other  
> thing over there - make it happen please!" with the minimum amount  
> of downloading/build time.

And simply having the solution available does not automatically mean  
that binaries will be. For instance Fink has their .info -> .deb  
wiring pretty soldered up already, but there's still no official  
binaries. I'm hoping that the MPAB project will at least get the  
building (with logging) part going. And that the MPWA project will do  
a better job of presenting these to the end-user. But I do believe  
it's going to use archives only, even if it could build "external"  
rpms and pkgs too.

>  If you look at FreeBSD ports, for example, where it could easily  
> be said that a "sophisticated audience" is the target (FreeBSD's  
> historical attempt to grab Linux mind/marketshare notwithstanding),  
> most users actually install software by typing "pkg_add -r gimp"  
> and letting the package management system do the rest, they don't  
> think of "cd /usr/ports/graphics/gimp; make all; make install" as  
> the most convenient/desirable way of doing it, and why should they  
> - the latter approach takes at least 10X the time to complete.

I think I used "portinstall -Pp" instead, because it did a better job  
at sorting out dependencies. But I could only get binaries to work  
for a while, once I had installed something from source it started  
throwing dependency errors like missing symbols and such until I had  
upgraded and rebuilt the other things to match. Installing KDE took  
all weekend, for instance. My limited understanding is that only the  
releases are offered as binaries (packages), and the interim software  
updates are only offered as source (ports).

>> The simplistic alternative here was to continue with the old  
>> archives,
>> and just enable "archivemode" to create them before any installation.
>
> Too simplistic, I think.  Again, installing a package is not simply  
> "de-archival" or we'd have taken that approach a long time ago.   
> Just grep through the current macports collection for all the ports  
> with pre/post install rules, for example.  What happens with  
> those?  What you're proposing would essentially require an end-user  
> to have the entire macports collection still on their hard drive so  
> that they could simply skip the "build step" and de-archive  
> instead, but still rely on the Portfiles for all the other install- 
> time information/procedures.  I suppose that's an incremental  
> improvement, but I still don't think it's what users want in the  
> final analysis.

The easy part of the archives was that it would still allow all the  
quirks from Portfile to continue working as before, whether it is  
about variants or post-activate actions or whatever else is buried in  
the Tcl. When translating these into packages, one would have to  
convert scripts and invent metadata to carry all features over and  
update the registry. It was my understanding that it was exactly the  
build step (and the accompanying need to install Developer Tools) was  
the one that most users was interested in skipping, not the  
installation of MacPorts. It could also be that Xcode Tools is 1000M  
and the Daily Tarball is 7M ?

And isn't so much of a suggested proposal, more like status quo... I  
prefer RPMS myself.

> What users *really* want is a graphical front end that talks to a  
> server somewhere "in the cloud" and pulls down an index of  
> available packages for their platform (which pkg_add -r also does  
> behind the scenes - if you're on Intel running FreeBSD-7, it grabs  
> packages just for that, vs what happens if you're running FreeBSD- 
> current on, say, a SPARC platform) then, because we're catering to  
> Mac users and not FreeBSD users, allows the user to just click on  
> the things they want, automatically pulling down dependencies and  
> running the post-install actions as necessary.

The architectures was taken care of, and so was the operating system.  
The problems that we had with the RPM packages was that they didn't  
separate between OS releases, and the platform variants wasn't  
recorded (no variants were). So you couldn't separate between  
freebsd6 and freebsd7 or between darwin7 and darwin8, which means  
that you could end up with the wrong binaries. This was hacked up  
into a bogus "Requires: org.darwinports.${os}${major}" dependency,  
but it wasn't (and still isn't) very pretty... Even if it does  
provide the (repodata) index of the packages and their dependencies.

The simplest workaround was to use different URLs for different  
releases, and keep the simple os-arch (or cpu-vendor-os-gnu)  
designator within each package repository.

>> This is similar to the "package format" that Slackware Linux uses.
>> Has a couple of shortcomings, but at least it is simple enough...
>
> Slackware's package manager also provides for install-time actions  
> and tracks dependencies.  There's more going on there than meets  
> the eye. :)

Not all _that_ much more, though. Just the "do-inst.sh" and the  
optional dependencies (Slackware Linux doesn't use them, but other  
distributions such as Vector Linux do), unfortunately all the  
metadata is mixed with the content which complicates e.g.  
compression. And the part where it is not all similar is where  
Slackware packages uses simple text files  and shell scripts, whereas  
the MacPorts archives would use fullblown Tcl programs (in form of a  
Portfile with support). Slackware packages and FreeBSD packages look  
rather similar, though ? Or at least they appeared that way to the  
Smart package manager... (http://smartpm.org/)

--anders