Supporting installing arbitrary port versions (was: Re: port "cask" -- installing prebuilt binaries)

[Jason Liu]

> Just looking at your idea to distribute all portfile versions, let's start
> with the fact that portfile syntax has evolved over time.
>

This is where portfile syntax itself can, and probably should, be
versioned. Maybe by incrementing PortSystem? i.e. PortSystem 1.3, 1.4, 2.0,
etc. Similar to how the HTML standard specification's version number has
changed over time, from HTML 2 all the way to the current HTML 5.

If MacPorts allowed the user to pick an arbitrary old Portfile to install,
> it is extremely likely that the user would experience an error.
>

>
If you are suggesting that we should not only distribute all old portfiles
> but also keep them all updated as needed, that would be a ludicrous amount
> of work that nobody would be willing to do.
>

This is where being able to restrict dependencies to specific version
ranges would be incredibly useful. If you could say that a portfile's
dependency requires a version >= 1.3.0 && <= 1.5.2, then you would not need
to update that old portfile once the dependency's portfile gets updated to
a version beyond 1.5.2.

Well you spoke of keeping all Portfile versions. So perhaps you're
> suggesting the user should be able to select from any past version of the
> port. That's slightly more reasonable than allowing the user to request any
> version of the software regardless of whether that version has ever been in
> the portfile, but still completely impossible and unsupportable as far as I
> can see so far.
>

Other PMSes like APT and YUM do exactly that. They keep all versions of
their package's spec files. How they do this is that their package
repositories have a different architecture than how MacPorts is currently
laid out.

[Note: What follows below is a fairly lengthy description of how another
package management system works. If you have no interest in such things,
here is probably a good place to stop reading.]

Allow me to take the Blender package in the Debian repo as an example. Here
is the entry for Debian Testing, which is the version of Debian that I use
(Debian Stable tends to be a bit too outdated for my taste):

https://packages.debian.org/bullseye/blender

If you look at the info on the right side of that page, one of the sections
is "Download Source Package blender:". There are three items listed:

   - [blender_2.83.5+dfsg-2.dsc]
   <http://deb.debian.org/debian/pool/main/b/blender/blender_2.83.5+dfsg-2.dsc>
   - [blender_2.83.5+dfsg.orig.tar.xz]
   <http://deb.debian.org/debian/pool/main/b/blender/blender_2.83.5+dfsg.orig.tar.xz>
   - [blender_2.83.5+dfsg-2.debian.tar.xz]
   <http://deb.debian.org/debian/pool/main/b/blender/blender_2.83.5+dfsg-2.debian.tar.xz>

Before I go into further detail about the three items above, another
interesting thing to take a quick look at is the actual folder for the
Blender package on the Debian repo:

http://deb.debian.org/debian/pool/main/b/blender/

There you will see the three files I listed above, and also many .deb
files. Those .deb files are basically pre-compiled archives of the package,
which have been built for individual architectures e.g. amd64, i386, arm64,
etc. Now, let me go through each of the three items.

The first item, the .dsc file, is a Debian source control
<https://wiki.debian.org/dsc> file. It is a text file that contains many
fields which would be quite familiar to anyone familiar with our portfiles.

The second item, the .orig.tar.xz file, is a copy of the upstream source
code. It's basically the equivalent of a MacPorts distfile, like what's
stored on https://distfiles.macports.org.

The third item, the .debian.tar.xz file, is the one that is the most
interesting. If we took the entire contents of this folder

https://github.com/macports/macports-ports/blob/master/graphics/blender/

and zipped it up into an archive, that would be the MacPorts equivalent to
the third item. If you extract the third item, you will find a debian/
folder that contains all of the spec files for the APT package of Blender.
In general, if you were to concatenate the two files "control" and "rules"
together into a single file, it would look a lot like our notion of a
portfile. The file called "control" will already look familiar: in fact,
this file is used to generate the .dsc file when the package gets built.
The "rules" file contains the equivalent of what we use in portfiles to
control the various phases of the MacPorts build process, e.g. pre-fetch,
post-patch, depends_lib, configure.args, post-destroot, etc.

Now, here is how an APT repository "keeps all Portfile versions". If you go
back and look at actual folder for the Blender package on the Debian repo:

http://deb.debian.org/debian/pool/main/b/blender/

you will see that there are multiple versions of Blender in that folder.
For each version, the three items I discussed are present, along with some
pre-compiled .deb archives. One of the most important ways in which an APT
or YUM repository differs from a MacPorts ports tree is this: what kinds of
data get downloaded to a client's computer during a sync operation. From
"man port" on my Mac:

sync
> Performs a sync operation only on the ports tree of a MacPorts
> installation, *pulling in the latest revision available of the Portfiles
> from the MacPorts rsync server*.
>

On an APT- or YUM-based Linux machine, none of the spec files for any
package get pulled down to a client machine during a sync operation.
Instead, what gets synced is a database that only contains metadata about
packages. This metadata is made up of entries from all of the .dsc files in
the entire Debian repository. So, when a user runs this command:

apt-get install blender=2.79.b+dfsg0-7

apt-get will first attempt to download the pre-compiled .deb archive. If a
package doesn't have a .deb available, then it will download all three
items and attempt to build from source. In this way, apt-get is basically
downloading the old portfile for Blender 2.79b when it gets the third item,
the .debian.tar.xz file.

On Sat, Oct 3, 2020 at 11:00 AM Ryan Schmidt <ryandesign at macports.org>
wrote:

> On Oct 3, 2020, at 06:05, Lothar Haeger wrote:
>>
>> Implementing all this would be a major project and I totally understand
>> that it's nothing to jump at without considering all the consequences and
>> outside factors.
>>
>
> It would be a major project if it were clear how it could possibly work
> and it were just a matter of doing it. But so far I have no idea how it
> could work.
>

As I said in my previous message, in the worst-case scenario it would
require a complete rewrite of MacPorts. Hopefully after I described how an
APT repo works, it's a bit more clear why I said that. Right now, MacPorts
has no concept of older and newer versions of a portfile. For example, in
my Blender port, there's no such thing as a "Portfile-2.82a",
"Portfile-2.83.0", "Portfile-2.83.1", etc. There's only a single
Portfile... the newest one. If I change the Portfile to a newer version of
Blender, I'm basically overwriting the same spec file for Blender, not
creating a brand new spec file for each new version of Blender, which is
how it is done for an APT or YUM repo.

-- 
Jason Liu


On Sat, Oct 3, 2020 at 11:00 AM Ryan Schmidt <ryandesign at macports.org>
wrote:

>
>
> On Oct 3, 2020, at 06:05, Lothar Haeger wrote:
>
> >> It's because, besides being an unimaginably large amount of work in
> rearranging our code to do it, I have absolutely no idea how it would be
> accomplished without providing the user with unlimited opportunities to
> create broken combinations of port versions, which would generate an
> unlimited number of bug reports that we would then need to respond to, and
> my goal in MacPorts is to reduce, not increase, the likelihood that users
> would find something broken or need to contact us to help troubleshoot it.
> >>
> >> If you have an idea for how it could be done without such problems
> arising, please open a new topic and describe it and we can talk through a
> few scenarios and see if it works.
> >
> > I do have some ideas (not mine, just looking at how it works elsewhere),
> but not the time to drive something of this size, to be honest. Basically,
> we'd need to
> > - keep and distribute all Portfile versions (including patches and
> files), not just the lastest
> > - add required versions to dependencies (can be version ranges e.g.
> >=1.2.3 or 2.* or 1.2.3-4.5.6)
> > - add support to install individual ports in into separate folders or
> give them individual names on install time (much like with the perl
> versions), so multiple versions of a port can be installed in parallel.
> > - add command line parameters to the port command so users can
> optionally specify a version to install (defaults to latest) and a pre/post
> fix to install location or file names or both (depends on how stuff gets
> implemented, of course)
> > - add support for dependency resolution including version information.
> So if two ports have different, non overlapping version requirements for a
> dependency, that dependency gets installed twice. Making sure each port
> then actually uses its matching version is probably one of the trickier
> parts here.
> >
> > Implementing all this would be a major project and I totally understand
> that it's nothing to jump at without considering all the consequences and
> outside factors.
>
> It would be a major project if it were clear how it could possibly work
> and it were just a matter of doing it. But so far I have no idea how it
> could work.
>
> Just looking at your idea to distribute all portfile versions, let's start
> with the fact that portfile syntax has evolved over time. A portfile from
> ten years ago can't even necessarily be parsed by today's MacPorts
> ("livecheck.type" used to be called "livecheck.check"; there used to be
> "cd" and "suffix" procedures that ports used to use) or it may behave
> differently than it did then ("platform" blocks used to behave like
> variants and their code was evaluated at the end of the portfile rather
> than where it's declared; the way that the arguments of startupitems and
> configure.env/build.env/etc need to be quoted was changed).
>
> Libraries change over time. Sometimes when a library changes it means that
> the other ports that use that library need to be patched. Obviously those
> patches were not in the old versions of the portfile that predate the new
> library, and patches for the current version of the software couldn't
> necessarily be used on old versions without being rewritten.
>
> Compilers and operating systems evolve. Xcode 12, for example, prohibits
> implicitly declared functions. We're adding patches to ports as fast as we
> can to fix these problems, but old versions of the portfile would not have
> those patches.
>
> If MacPorts allowed the user to pick an arbitrary old Portfile to install,
> it is extremely likely that the user would experience an error.
>
> If you are suggesting that we should not only distribute all old portfiles
> but also keep them all updated as needed, that would be a ludicrous amount
> of work that nobody would be willing to do.
>
>
> >> I'm speaking of the user being able to specify an arbitrary version.
> >
> > So do I.
>
> Well you spoke of keeping all Portfile versions. So perhaps you're
> suggesting the user should be able to select from any past version of the
> port. That's slightly more reasonable than allowing the user to request any
> version of the software regardless of whether that version has ever been in
> the portfile, but still completely impossible and unsupportable as far as I
> can see so far.
>
>
> >> I consider it a feature, not a bug, that we offer multiple ports for
> different versions of perl, php, python, ruby, gcc, clang. It enables the
> user to install multiple versions of those languages that can all be active
> and available at the same time. If the user has one script that works with
> python 3.8 and another that requires python 3.6, no problem. If they have
> one web site that works with php 7.4 and another that needs php 7.2, no
> problem. If we only had a single python or php port that only let the user
> choose a single version to install at a time, that would not be possible.
> >
> > I totally agree that this is a great feature. Only it's limited to a few
> versions of a few ports due to the way it's being implemented. Supporting
> this for more ports and version does not scale well, twice as many
> ports/versions cause twice as much work. With thousands of ports in the
> tree the current approach will always be limited to a handful of ports.
>
> Exactly: it's a lot of work to implement it correctly, which is why it's
> only done for a few ports where it's really useful.
>
> > If this was implemented as general features to a) install any port
> version and b) install several instances of any port, we'd get the same for
> each and every port/version out of the box.
>
> I have no idea how the manual labor that we currently expend on
> implementing this feature selectively could be automated or generalized to
> all ports.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macports.org/pipermail/macports-dev/attachments/20201004/f3fada5a/attachment-0001.htm>