Binary Packages (again)

[Jordan K. Hubbard]

On Mar 21, 2011, at 2:20 AM, Anders F Björklund wrote:

> Jordan K. Hubbard wrote:
> 
>> On Mar 13, 2011, at 4:11 AM, Anders F Björklund wrote:
>> 
>>> With "real" packages, I mean the ones where the metadata
>>> (header, control) is separate from content (payload, data)
>>> Such as .deb and .rpm (or the proprietary .pkg), but unlike
>>> .tgz and friends - the ones we are just calling "archives".
>> 
>> I'm not sure that I understand how the distinction between "real" and sub-functional packages is a function of where the metadata is stored.  My understanding of the "archive" format has always been that it proves some, but not enough, of the +CONTENTS file metadata necessary to really represent, from a purely functional standpoint, an installable package.  If it had more such metadata, at least as much as reasonably expected by the FreeBSD pkg_install tools (and I know that I still owe folks a port to Darwin of those tools - got distracted by other things), then MacPorts archive files would be no more or less installable than arbitrary binary packages on FreeBSD, and that meets the criteria of "good enough", does it not?  
> 
> Yeah, if MacPorts did that it would be more than destroots...
> FreeBSD/Slackware/ArchLinux/etc tarballs are "packages", sure.
> But if those binaries still require a full ports tree to get
> the needed information, it's not more than just "archives".

I'm still not sure where the requirement for the "whole ports tree" comes into this?  I mean, if you really needed something from inside the Portfile directly, I suppose you could also stick that in the tarball and throw it away later - that is already the default behavior for any file extracted by the tarball that doesn't appear in the "packing list" - it's assumed to be metadata.  You could then have pre/post install scripts which scraped the Portfile in its temporary location for whatever data you needed out of it then just let the pkg tools clean it up afterwards.

>> Yes, in /tmp - it was a lousy design decision and I regret it to this day (I had some weird notion of "transactional installs" which never ended up being truly transactional anyway) but on most Mac OS X systems that's still the same volume as the target file, so at least you just get a temporary unpack followed by a rename of each file/dir into place and it's the same amount of disk space, transient or otherwise, either way in the end.
> 
> No, it's not... You need to download the entire content,
> just to get at the metadata ?

Actually not - the pkg tools actually extract the +CONTENTS file directly from the tarball (tar even has an option for "look for the following file and then STOP when you have found it" which pkg_add uses) and pkg_create sticks this file right at the beginning so that a minimum amount of the tarball needs to be decompressed.  This is because one of the options in +CONTENTS is "extract in place" for really big packages like X11, and the pkg_add tool needs to know this early on.

> And also unpack all of it, probably using the slowest decompression available: bzip2.
> So it both takes more disk space, and much more memory too.

What I meant was that if you do the following:

1. Extract a bzip2 compressed tarball into / directly

vs

2. Extract a bzip2 compressed tarball into /tmp and then mv each file into /

The two are essentially equivalent in terms of disk space because / is on the same volume as /tmp on most MacOSX boxes.

- Jordan