Package managers and package versioning

[Jason Liu]

>
> It's my understanding, for example, that Homebrew allows the user to
> request to install the git master of a particular Homebrew formula, in
> effect allowing the user to (try to) install a version of the software that
> was never previously attempted by the formula's creator. This is what I was
> afraid was part of what was being proposed, and something I would not
> support adding to MacPorts.
>

I'm not that familiar with Homebrew, so this is the first time I'm hearing
about this sort of capability. None of the other PMSes that I have
experience with can do this. The idea of being able to (try to) install a
version of a package that doesn't even exist in a repository's metadata
database sounds insane to me.

If we continue the liberal approach with dependencies, then we begin from
> the assumption that a port can use any version of a dependency. Let's say
> libpng declares it can use any version of zlib. Then years later zlib 2 is
> released and it is incompatible, breaking all versions of libpng we've
> published. There's now a lot of work to go edit each previous libpng to
> indicate that it requires zlib < 2.
>

#!/bin/bash
for f in libpng/Portfile-*; do
    sed -i -e s/(port:zlib.*)$/\1 < 2/ $f
done

I'm probably missing some backslashes, but I think you probably get the
idea. That doesn't seem like that much work to me.

Alternatively, suppose we use the conservative approach and declare that
> libpng requires zlib < 2. Or zlib < 1.3. Or zlib < 1.2.12. We have no way
> to know in what future version of zlib, if any, incompatibility might
> arise. Suppose we declare compatibility with zlib < 2, and years later zlib
> 2 comes along and maintains perfect backward compatibility with zlib 1.
> Then our libpng would continue to use the old zlib 1, even though it would
> work with zlib 2, thus making users use outdated dependencies unnecessarily
> and possibly exposing them to bugs that have already been fixed in newer
> versions. And the solution then is to do a lot of work to go edit each
> previous libpng to update its zlib compatibility list.
>

A sed statement wrapped inside a for loop would probably also help in this
case.

Either way, it's signing all port maintainers up for a lot of work. (Many
> port maintainers already don't do the currently expected amount of work to
> maintain their ports.) And all for the benefit of other ports being able to
> declare that they require an old version of libpng. We did have this
> problem once before, when some old ports were not compatible with libpng
> 1.4 when it came out. Rather than introducing a libpng12 compatibility
> port, we did the right thing and fixed other software to be compatible with
> libpng 1.4. If we had used a libpng12 port instead, we likely would never
> have noticed if and when those ports gained libpng 1.4 compatibility and
> would be forcing libpng12 on users unnecessarily for years to come.
>

However, I feel like this would be a good example of why having multiple
versions of a package's portfile available could be beneficial. Let's say
libpng 1.4 came out and it broke a bunch of old ports. If previous versions
of the portfile were available, like what I showed in my example of the
Blender Debian package, then it would be very easy to simply remove the
libpng 1.4 portfile from the repository, and all of the ports that were
broken would suddenly be fine again, because they would just go back to
using whichever portfile for libpng was already there. No need to increase
the epoch inside the portfile to try to back-rev to an older version of
libpng, no need to create a libpng12 compatibility port. And the port
maintainers would be able to go right back to working on fixing ports
locally on their test machines, without having to worry that the actual
global ports tree had been broken for users. (By the way, this is why most
flavors of Linux have multiple release channels, typically "stable",
"testing", and "unstable". Each release channel has its own PMS repository,
to avoid the situation where the global repository that users use gets
polluted due to a package incompatibility.)

And all for the benefit of other ports being able to declare that they
> require an old version of libpng.
>

If there's really such little benefit, then why are we even having such a
discussion? And why are there other PMSes which have this ability? Because
they enjoy torturing their package maintainers with lots of busy work?
Somehow I doubt it.... They wouldn't have gone to all the trouble of adding
such a feature unless it was able to reduce the amount of work they had to
do in some way.

This is one place where MacPort's activate/deactivate would really shine:
>> If someone wanted to install an old package, MacPorts would first
>> deactivate all of the dependencies that were too new, and then install the
>> old versions of the dependencies that are compatible with the old package.
>>
>
> If all versions of a port install to the same location on disk as they do
> currently and only one version of a port can be active at a time, then your
> proposal here -- "deactivate all of the dependencies that were too new" --
> will break all the ports that depend on the newer dependencies.
>

Yes, that is precisely what I am proposing: that doing so would break all
ports that depend on the newer dependencies. This is exactly what happens
in PMSes like APT and YUM. And very similar to what happens if you were to
do a "port deactivate libpng", the PMS will provide a helpful warning
message telling you that the following list of packages will be broken if
you proceed... do you still wish to continue? [y/N].

I realize that there are probably many people screaming at me through their
monitors right now saying, "That's a TERRIBLE idea! Why would we want to
allow users to break a bunch of stuff using our system?!" Other PMSes will
warn you when you are about to do something like that, but the fact is that
they will still allow you to do it if you insist on continuing. Again, I
point out that this is where MacPorts' activate/deactivate system could
really shine, because MacPorts still retains on local disk the other
version that was already installed, and you can switch back and forth
between versions using "port activate". This is still much better than APT,
which will simply uninstall the newer version of the dependency and replace
it with the older version (i.e. you can only have one copy of a package
installed on the system at one time).

For us, the single product we are providing is MacPorts, which has a very
> broad focus: ten thousand software packages intended to work together. That
> goal isn't always met: sometimes updating one port breaks another, until we
> fix it. But treating each of those ten thousand ports as a separate
> top-level product that needs its own world of dependencies would be an
> unfathomable disaster.
>
<snip>

> Don't forget also that separate libraries would be loaded into memory
> separately. If you have a zillion copies of gettext and libpng installed,
> one for each port that uses them, and then you want to use something huge
> like Qt or WebkKt, then you're loading hundreds of copies of gettext or
> libpng into memory.
>

But isn't this precisely what NPM does? Apple's take (and possibly NPM's
too) would be to say, "storage and memory are cheap these days". I don't
know where all this supposed cheap stuff is available for sale, but I
certainly don't have the money to just go out and upgrade the storage and
memory on all of my computers.

I hope it's clear by now, but just in case we're still in "nothing is
obvious" territory, let me clearly state that my personal preferences
generally lean more towards the side of the *nix-like philosophy of
"sharing/reusing as many resources as possible is better". However, I think
you probably have to admit one thing about NPM is that its packages are
very self-contained.

I thought it was being proposed that all versions of a port would install
> to different locations on disk so that many could be active simultaneously,
> which would avoid the one problem while causing many new ones, most of
> which we haven't begun to discuss yet.
>

Let me try an example.
>
> Suppose I have zlib installed at:
> /opt/local/port/zlib/1.2.11/{bin,include,lib}
>
<snip>

> So I hope this npm-style hierarchy is not what anyone is proposing.
>

No, I certainly hope nothing that I had said before was giving the
impression that this is what I was suggesting, and in my mind this would be
a bad idea. However, I think it's impossible to deny that all of the PMSes
mentioned so far, APT, YUM, NPM, Yarn, etc. have all been quite successful,
and they all give users the ability to install specific versions of
packages.

-- 
Jason Liu


On Sun, Oct 4, 2020 at 5:34 PM Ryan Schmidt <ryandesign at macports.org> wrote:

>
>
> On Oct 4, 2020, at 14:09, Jason Liu wrote:
>
> >> Where by "this" you again mean the ability to specify a
> previously-available version, not an arbitrary version that was never in a
> portfile.
> >
> > Yes, of course. If you look at the folder for the Blender package on the
> Debian repo
> >
> > http://deb.debian.org/debian/pool/main/b/blender/
> >
> > from my previous example, there are many versions of Blender that were
> skipped. For instance, they skipped version 2.82 and went from 2.79b right
> to 2.82a. And they skipped versions 2.83.0-2.83.4, and went from 2.82a
> right to 2.83.5. Obviously, there would be no way for a user to try to
> install Blender version 2.83.2 using apt-get, because that version was
> never packaged (i.e. spec files don't exist) for that particular version of
> Blender.
>
> Let's try not to say "of course". Nothing is obvious in this discussion.
> Let us be very clear at every step what we are proposing so that we can
> have a meaningful discussion.
>
> It's my understanding, for example, that Homebrew allows the user to
> request to install the git master of a particular Homebrew formula, in
> effect allowing the user to (try to) install a version of the software that
> was never previously attempted by the formula's creator. This is what I was
> afraid was part of what was being proposed, and something I would not
> support adding to MacPorts.
>
>
> > The examples were meant to illustrate the fact that it's possible to
> restrict the dependency versions when installing an old package. By
> specifying maximum versions for all of a package's dependencies (by using
> something like <=2.4.0 or <2.4.0), it would guarantee that the old package
> would never have to be built against new dependencies, because if a user
> were to install the old package, it would also install older versions of
> the dependencies.
>
> I understand that it would be possible to define a syntax for specifying
> dependency versions and that MacPorts base could be enhanced to parse it
> and even to find and download the right version of the portfile and to
> install different versions of ports to different locations.
>
> The problem is that if you are currently compatible with the latest
> version of a dependency, you do not know in advance whether you will be
> compatible with the next version of that dependency.
>
> So you can either be conservative or liberal. MacPorts has tended to be
> liberal in these matters, assuming that a port can build on any
> architecture, any OS version, with any compiler. Once it is discovered that
> it can't build on some subset of those, restrictions can be added.
> supported_archs can be used to rule out incompatible architectures.
> compiler.blacklist can be used to rule out incompatible compilers. Ports
> can indicate which OS versions they're compatible with, though not as
> elegantly as I would like.
>
> If we continue the liberal approach with dependencies, then we begin from
> the assumption that a port can use any version of a dependency. Let's say
> libpng declares it can use any version of zlib. Then years later zlib 2 is
> released and it is incompatible, breaking all versions of libpng we've
> published. There's now a lot of work to go edit each previous libpng to
> indicate that it requires zlib < 2.
>
> Alternatively, suppose we use the conservative approach and declare that
> libpng requires zlib < 2. Or zlib < 1.3. Or zlib < 1.2.12. We have no way
> to know in what future version of zlib, if any, incompatibility might
> arise. Suppose we declare compatibility with zlib < 2, and years later zlib
> 2 comes along and maintains perfect backward compatibility with zlib 1.
> Then our libpng would continue to use the old zlib 1, even though it would
> work with zlib 2, thus making users use outdated dependencies unnecessarily
> and possibly exposing them to bugs that have already been fixed in newer
> versions. And the solution then is to do a lot of work to go edit each
> previous libpng to update its zlib compatibility list.
>
> Either way, it's signing all port maintainers up for a lot of work. (Many
> port maintainers already don't do the currently expected amount of work to
> maintain their ports.) And all for the benefit of other ports being able to
> declare that they require an old version of libpng. We did have this
> problem once before, when some old ports were not compatible with libpng
> 1.4 when it came out. Rather than introducing a libpng12 compatibility
> port, we did the right thing and fixed other software to be compatible with
> libpng 1.4. If we had used a libpng12 port instead, we likely would never
> have noticed if and when those ports gained libpng 1.4 compatibility and
> would be forcing libpng12 on users unnecessarily for years to come.
>
>
> > This is one place where MacPort's activate/deactivate would really
> shine: If someone wanted to install an old package, MacPorts would first
> deactivate all of the dependencies that were too new, and then install the
> old versions of the dependencies that are compatible with the old package.
>
> If all versions of a port install to the same location on disk as they do
> currently and only one version of a port can be active at a time, then your
> proposal here -- "deactivate all of the dependencies that were too new" --
> will break all the ports that depend on the newer dependencies.
>
> I thought it was being proposed that all versions of a port would install
> to different locations on disk so that many could be active simultaneously,
> which would avoid the one problem while causing many new ones, most of
> which we haven't begun to discuss yet.
>
>
> >> Despite the bloat it might work for that use case, but some npm module
> developers unnecessarily restrict the versions of dependencies they're
> compatible with, meaning that you get an unnecessarily outdated version of
> that dependency, even though a newer version of the dependency would work
> and might fix some bug you're experiencing. And this ballooning of
> dependencies can use a tremendously larger amount of disk space than if
> just a single compatible copy of each dep had been installed. We already
> have users complaining about unnecessary dependencies and unnecessary disk
> usage; this would just make it worse.
> >>
> >> And do you really want to install a separate copy of gettext for each
> dependency that uses it?
> >
> > Well, let's be frank here. This is the "Apple way". Other than the
> system frameworks, Apple has always strongly encouraged that software keep
> all of its bits and pieces inside of its application bundle in the
> /Applications folder. This has traditionally meant that multiple apps would
> have their own copy of the same gettext library in each of their
> application bundles; because Apple has never said "hey developers, put all
> of your shared libraries in this /usr/local or /opt/local folder". Instead,
> they've always gone the route of "each and every app only gets to play in
> its own sandbox; you should bring all of your own toys (i.e. libraries) to
> your own sandbox... no sharing of toys! (other than the ones Apple provides
> and controls, i.e. system frameworks)"
> >
> > I'm not saying that I agree with Apple's viewpoint, but I do find it a
> bit odd that users would use a *nix-like argument of "share as much as
> possible to reduce bloat" to complain about something (MacPorts) that lives
> on an Apple Corporation-made system, where "share as little as possible" is
> the norm. MacPorts is already a shining example of sharing and reusing the
> same libraries on a system that highly discourages such behavior.
>
> What you say about the "Apple way" of doing application bundles is true,
> and it in no way follows that we should do the same in MacPorts.
>
> Consider an application like Firefox, developed by an organization, using
> various open source libraries, which are all included in the application
> bundle. But that's it. There's only one copy of each dependency because the
> developers know that they are creating a single product and that all the
> dependencies need to work together. They can hold back updating certain
> dependencies if they know it will cause problems for other libraries they
> use. They can exclude manpages, programs or other files they don't need and
> distribute only what's needed for their specific use case within Firefox.
> They have a narrow focus.
>
> For us, the single product we are providing is MacPorts, which has a very
> broad focus: ten thousand software packages intended to work together. That
> goal isn't always met: sometimes updating one port breaks another, until we
> fix it. But treating each of those ten thousand ports as a separate
> top-level product that needs its own world of dependencies would be an
> unfathomable disaster.
>
> Installing all ports to a different top-level prefix for that port name
> and version, which is what I thought we were talking about initially, is
> slightly less disastrous. For example, if we had gone the other way with
> the libpng example above, we could have prefixes
> /opt/local/port/libpng/1.2.57 and /opt/local/port/libpng/1.6.37, each of
> which would contain the bin, include, lib, etc. directories for that
> version of libpng. Then the majority of ports that are fine with 1.6 could
> use that and those that need 1.2 could use that, and then that would only
> be two copies of libpng installed instead of a thousand.
>
> Don't forget also that separate libraries would be loaded into memory
> separately. If you have a zillion copies of gettext and libpng installed,
> one for each port that uses them, and then you want to use something huge
> like Qt or WebkKt, then you're loading hundreds of copies of gettext or
> libpng into memory.
>
>
> >> But you haven't addressed how you could build historical "Portfiles" on
> a newer OS or compiler version when they would not have been written to
> account for the new restrictions imposed by that newer OS or compiler
> version, and we go back to my supposition that a portfile maintainer would
> be asked to keep updating and patching ever old versions of their portfile.
> >>
> >> How would you address the "implicit declaration of function" problem I
> mentioned? A new version of Xcode, 12, has come along and shown us that a
> zillion of our ports, such as php, do not include the headers that declare
> the functions they are using. We must fix this for compatibility with
> future Macs.
> >>
> >> This is just an example of the type of problem that comes up all the
> time. Yosemite required tons of patches because many build systems
> misinterpreted 10.10 as 10.1. macOS 11 is requiring tons of patches because
> many build systems expected the macOS major version to remain at 10
> forever. Many build systems need patches now for ARM support. etc. etc. etc.
> >
> > In this particular case, I believe that the Xcode version, and even the
> macOS version, should be counted as just another type of dependency... one
> that happens to be required in every single portfile. If these (required)
> variables also had the ability to specify version ranges, we could
> effectively cap each version of a portfile with a maximum compatible
> version of macOS and Xcode, which could be raised as compatibility patches
> are added for newer macOSes and Xcodes.
> >
> > macos_version_compatibility <=10.15.7
> > (I prefer to use Darwin version)
> >
> > darwin_version_compatibility <=19
> > (this portfile would be compatible up to Catalina, but not yet
> compatible with macOS 11)
> >
> > xcode_version_compatibility >=8.2.1 && <=12beta2
> > (or whatever)
>
> Agreed we should have a way to specify OS version compatibility with a
> variable. It is a long-standing wish. There is a ticket open about maybe
> using the currently-unused platforms variable for this purpose. Many ports
> do indicate OS compatibility but with manually-written code blocks.
>
> We already have the supported_archs variable for indicating architecture
> support.
>
> Xcode version compatibility has historically not been so interesting; it's
> usually the compiler and SDK version that's relevant. We already support
> restricting which compilers can build a port, using compiler.blacklist. I'm
> working on something similar for SDK versions. We do already have the
> xcodeversion portgroup for restricting Xcode versions, but it is not
> usually necessary and it is old and does not account for the fact that the
> command line tools are now separate from Xcode and that one might want a
> way to specify "Xcode < 12 or the command line tools corresponding to Xcode
> < 12". For completeness with other version checking variables this
> capability probably belongs in MacPorts base, but the same can be said for
> many of the portgroups.
>
>
> > The benefit of this would be as follows. Let's say that we went to a
> system of having one portfile for each version of a particular package.
> Let's also say that "somepackage" was currently on version 2.4.3, and it is
> not compatible with macOS 11. If the upstream authors of "somepackage" were
> to add patches that made it compatible with macOS 11 and released it as
> 2.4.4, then the new portfile for 2.4.4 would increment
> darwin_version_compatibility <=20. However, the portfile for 2.4.3 wouldn't
> need to be changed at all, because it wasn't, and never will be, compatible
> with macOS 11. If a port maintainer were to come along and backport the
> macOS 11 patches into the 2.4.3 port, then they could update the 2.4.3
> portfile to also be compatible with darwin <=20. A similar concept applies
> to Xcode version, ARM support, etc.
> >
> > Thus, the idea of keeping a separate portfile for each version of a
> package would answer the question of "how you could build historical
> "Portfiles" on a newer OS or compiler version", because the answer is: you
> never would. Once a "Portfile-2.4.3" was pegged to a maximum of Catalina,
> it would potentially never need to be updated again, because it is not
> compatible when macOS 11 gets released. Our new "port" command would be
> able to indicate to users on a macOS 11 machine that only "Portfile-2.4.4"
> was available, because the old historical "Portfile-2.4.3" isn't compatible
> with macOS 11.
> >
> >> This is all work that we are here volunteering to do to improve our
> current port collection, but asking everyone to retroactively also fix
> these problems in all older versions would not be acceptable. We don't have
> a sufficient number of contributors to keep our current collection of ports
> up to date, so we certainly don't have enough contributors to keep all
> previous versions of our collection working as well.
> >
> > This is why it would be important in a "new" MacPorts to be able to cap
> old versions of a portfile with a maximum compatible version of stuff.
> ("stuff" meaning dependencies, macOS version, Xcode version, etc.) This
> would allow older versions of portfiles to slowly fade into obsolescence as
> their maximum compatible versions of dependencies become more and more
> out-of-date. There would be no need to keep retroactively updating them
> once they've been capped with maximum compatible versions.
> >
> > If compatibility patches can be backported, then great, but if they
> can't be for any reason, then that's fine too, the old versions of the
> portfiles will continue to work for older versions of macOS/Xcode. But in
> my opinion that would be a much better situation than what exists now,
> where there's only a single Portfile (and a single version of the package)
> for all users on all versions of macOS. However, I once again reiterate
> that I also believe it would require a massive amount of effort to
> implement this new regime.
> >
> >> You might even have to compile each of those copies from source, since
> each copy is going to a different location, and MacPorts binaries are not
> designed to be relocatable. Making our binaries relocatable is something
> I'm interested in but that is also a large challenge.
> >
> > MacPorts binaries aren't designed to be relocatable, because MacPorts,
> in my opinion, operates under a very *nix-like philosophy of sharing and
> reusing resources. This is also why many macOS apps are relocatable:
> because they don't share resources with anything else, and thus can play in
> their own sandbox. Regardless of where you relocate this sandbox to, there
> is very little chance of the app breaking when you move it, because all of
> its toys (like the gettext discussed above) move along with it. In many
> cases, sharing/reusing resources and relocatability are direct trade-offs
> and in opposition to one another. (Obviously, things like @rpath,
> @executable_path, @loader_path, etc. were designed to help break this
> linkage/trade-off.)
>
> All of this is true and none of it addresses the point I was making.
>
> In this paragraph I was responding to the comparison to npm, a package
> manager using a hierarchical dependency scheme, or whatever you want to
> call it, and why it would not work well for us, as far as I can tell.
>
> Let me try an example.
>
> Suppose I have zlib installed at:
> /opt/local/port/zlib/1.2.11/{bin,include,lib}
>
> Now suppose I want libpng and it installs to:
> /opt/local/port/libpng/1.6.37/{bin,include,lib}
> It depends on zlib, and so as not to conflict with anyone else's zlib it
> installs its own copy:
> /opt/local/port/libpng/1.6.37/port/zlib/1.2.11/{bin,include,lib}
>
> Now I want to install gd2 and it installs to:
> /opt/local/port/gd2/2.3.0/{bin,include,lib}
> It depends on zlib so it installs:
> /opt/local/port/gd2/2.3.0/port/zlib/1.2.11/{bin,include,lib}
> It also depends on libpng so it installs:
> /opt/local/port/gd2/2.3.0/port/libpng/1.6.37/{bin,include,lib}
> and:
>
> /opt/local/port/gd2/2.3.0/port/libpng/1.6.37/port/zlib/1.2.11/{bin,include,lib}
>
> We now have lots of extra disk usage on the user's system, lots of extra
> memory usage for loading all those redundant libraries, and we have
> problems regarding how did the above get installed: there are a few
> possibilities:
>
> 1. The user had to build everything from source because we don't have
> binaries. That's a step back from where we are now. Users like binaries.
> 2. We had to build each of those permutations of libpng, zlib, and
> everything else on our build server and distribute each of those specific
> binaries that are tied to those locations. That's a step back from where we
> are now in terms of wasting server build time and disk space.
> 3. We build just one libpng, zlib, etc. on our build server and we have to
> figure out how to make the packages relocatable. This is an interest of
> mine but we've done nothing yet to try to make this work. Note that just
> the task of identifying all Mach-O files and fixing the install_names and
> library linkage is easy; dylibbundler shows us it can be done. The hard
> part is figuring out where else the prefix might be baked in and how to
> solve it. Even MacPorts base itself is not relocatable; see
> https://trac.macports.org/ticket/56204 for why this can be a tougher
> problem than a simple find-replace in the destrooted files.
>
> So I hope this npm-style hierarchy is not what anyone is proposing.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macports.org/pipermail/macports-dev/attachments/20201005/60b43118/attachment-0001.htm>