[GSoC][Binaries support] Architecture

[Anders F Björklund]

Jordan K. Hubbard wrote:

>>> 1. pkg needs to be able to instantiate a tcl runtime environment that can pull in all of the port 1.0 Tcl files such that the post-install (and any other special post installation procedures) can run with the full environment, e.g. commands like "ui_msg" will be present and variables like ${name} will expand properly.
>> 
>> Maybe this is better handled by "port -b", if it needs the full environment anyway ?
> 
> I'm not sure what "port -b" is, but I'll assume it's "install from binary package."  

     -s       source-only mode (build and install from source, ignore all
              binary archives, do not create/recreate binary archives) (only
              applies when archive mode is enabled)

     -b       binary-only mode (build and install from binary archives, ignore
              source, abort if no archive present; do not create/recreate
              binary archives from source) (only applies when archive mode is
              enabled)

Basically it would make port(1) fail, rather than build from source...
If you don't have Xcode installed, that's probably the result anyway.

> Let me tell you a little story that, I think, will also make my point nicely: [...]

> My little story doesn't actually help us to decide whether to express package installation commands as Tcl procedures or a mixture of shell commands and property settings which are consumed by some pkg tool, of course, but it does demonstrate that you can actually re-converge the evolution of these two separate problems by brute-force if you really want to. :-)

I think Fink is already doing "create packages to be installed by a
pkg tool" (in that case dpkg), and that we already "tried" rpm/xpkg ?

It seems that there is no such tool existing that would be accepted.
But _maybe_ the MP archives could be extended enough to create one...

>>> 2. We need to create a different set of "installation commands or scripts" that are associated with a port and specifically designed to run at post-installation time, e.g. instead of a post-install command in Tcl, we would have some sort of "install script" that is bundled with the port and merely relies on a few key environment variables to be set in order to run.  Then all 115 ports which currently declare post-install need to be modified such that they use the external script instead, and the port(1) system needs to stop running (or looking for) the post-install command and instead chain to the script so that "port install blah" still does the same thing as "pkg install blah".
>> 
>> This would probably be better, either @exec/@unexec or something much more "tailored" ?
> 
> If you're willing to "sin" from a security standpoint by allowing arbitrary @exec/@unexec operations to occur, then you can just skip the tailoring and leave things open-ended (or defined by policy) since you have no realistic hope of controlling what packages do if that's your sin of choice.  Of course, If you think you can actually express the full set of installation operations as a set of properties which folks who are used to "make install" can also realistically cope with then you're a smarter / better man than myself and I wish you all the luck in the world, seriously, because I will be observing with great interest! :-)

It's open-ended, sure. But it would be shell scripts rather than Tcl...
(just so that you wouldn't need a Tcl interpreter to run them, that is)
If choosing a set of operations, it would be need to be based on ports.
(so it would be possible to convert any existing scripts over to those)

>>> Option #2 is the simplest in terms of not having to instantiate a full Tcl environment instead of pkg(1), but you'd then have to change the way all ports/pkgs are installed.
>> 
>> I think it would be best to *not* have a Tcl interpreter (and thus ignore +PORTFILE), as long as it can interact with the registry. It should have "enough" information in the +CONTENTS, even if it requires modifying those 115 out of 7870 ports that require it. Otherwise there isn't much of a difference between port(1) and pkg(1) ? But you would still need to re-implement all the other bits - like compression, signatures, depsolving, etc. etc.
> 
> As long as the registry can be interacted with by @commands in the +CONTENTS file as well, sure.  Say an @exec command creates a configuration file on the package's behalf at install time - I need to be able to then declare what that file name was to the registry even though it wasn't in the original packing list.  Similarly, an @unexec might want to remove something.  Keeping the registry in sync is one of the MacPorts-specific challenges that actually having a registry database entails.  I don't know if the FreeBSD folks ever added one (I suspect they did), but they must have figured this out in the pkg tools if so.

I've only used the portupgrade tools, which re-create the database after changes... So it keeps a separate portsdb and pkgdb, that needs updating everytime the ports tree or installed pkg do. So it's still the same flat-file format used for "interchange".

>>> I honestly have no preference at this point given that Landon and I managed to argue ourselves into a stalemate over the question of port runtime versioning, how to deal with different architectures (there's not just a Tcl piece, but also Pextlib) and blah blah blah, and in hindsight I think we vastly overcomplicated the problem in our minds.  The actual implementation, particularly now that we've learned that the port system never versions (we're still at 1.0 how many years later? :)) and there's really only one architecture working caring about now (x86_64), of option #1, should you choose to go that way, should be pretty easy to get to "good enough" status.
>> 
>> MacPorts spent a couple of years discussing the optimal logging format, instead of actually implementing it. So that seems rather common. Hopefully at least archives can be delivered this time, before moving on to packages.
> 
> I'm still not sure I understand the distinction if archives become "smart enough" to also be packages.  I think that's also a good thing.

I meant "archives without metadata", like if just tarring up the destroot ? That's still enough for "port unarchive" to work with, and it would still neatly avoid the "port build" phase.

If the MPAB server can provide archives, and "port archivefetch" can download them, that's still a big improvement even if the archives don't have metadata and can't be used on their own.

I think that a pkg(1) tool could be a great addition. But maybe without a "stable" branch and with automatic upgrade of all dependencies, it would be "good enough" to build the binaries.

--anders