Package managers and package versioning

[Jason Liu]

>
> Where by "this" you again mean the ability to specify a
> previously-available version, not an arbitrary version that was never in a
> portfile.
>

Yes, of course. If you look at the folder for the Blender package on the
Debian repo

http://deb.debian.org/debian/pool/main/b/blender/

from my previous example, there are many versions of Blender that were
skipped. For instance, they skipped version 2.82 and went from 2.79b right
to 2.82a. And they skipped versions 2.83.0-2.83.4, and went from 2.82a
right to 2.83.5. Obviously, there would be no way for a user to try to
install Blender version 2.83.2 using apt-get, because that version was
never packaged (i.e. spec files don't exist) for that particular version of
Blender.

Those links again do not explain how to overcome the challenges of building
> an old portfile against new dependencies or on new operating systems or
> compilers.
>

The examples were meant to illustrate the fact that it's possible to
restrict the dependency versions when installing an old package. By
specifying maximum versions for all of a package's dependencies (by using
something like <=2.4.0 or <2.4.0), it would guarantee that the old package
would never have to be built against new dependencies, because if a user
were to install the old package, it would also install older versions of
the dependencies.

This is one place where MacPort's activate/deactivate would really shine:
If someone wanted to install an old package, MacPorts would first
deactivate all of the dependencies that were too new, and then install the
old versions of the dependencies that are compatible with the old package.

Despite the bloat it might work for that use case, but some npm module
> developers unnecessarily restrict the versions of dependencies they're
> compatible with, meaning that you get an unnecessarily outdated version of
> that dependency, even though a newer version of the dependency would work
> and might fix some bug you're experiencing. And this ballooning of
> dependencies can use a tremendously larger amount of disk space than if
> just a single compatible copy of each dep had been installed. We already
> have users complaining about unnecessary dependencies and unnecessary disk
> usage; this would just make it worse.
>

> And do you really want to install a separate copy of gettext for each
> dependency that uses it?
>

Well, let's be frank here. This is the "Apple way". Other than the system
frameworks, Apple has always strongly encouraged that software keep all of
its bits and pieces inside of its application bundle in the /Applications
folder. This has traditionally meant that multiple apps would have their
own copy of the same gettext library in each of their application bundles;
because Apple has never said "hey developers, put all of your shared
libraries in this /usr/local or /opt/local folder". Instead, they've always
gone the route of "each and every app only gets to play in its own sandbox;
you should bring all of your own toys (i.e. libraries) to your own
sandbox... no sharing of toys! (other than the ones Apple provides and
controls, i.e. system frameworks)"

I'm not saying that I agree with Apple's viewpoint, but I do find it a bit
odd that users would use a *nix-like argument of "share as much as possible
to reduce bloat" to complain about something (MacPorts) that lives on an
Apple Corporation-made system, where "share as little as possible" is the
norm. MacPorts is already a shining example of sharing and reusing the same
libraries on a system that highly discourages such behavior.

But you haven't addressed how you could build historical "Portfiles" on a
> newer OS or compiler version when they would not have been written to
> account for the new restrictions imposed by that newer OS or compiler
> version, and we go back to my supposition that a portfile maintainer would
> be asked to keep updating and patching ever old versions of their portfile.
>

How would you address the "implicit declaration of function" problem I
> mentioned? A new version of Xcode, 12, has come along and shown us that a
> zillion of our ports, such as php, do not include the headers that declare
> the functions they are using. We must fix this for compatibility with
> future Macs.
>

>
This is just an example of the type of problem that comes up all the time.
> Yosemite required tons of patches because many build systems misinterpreted
> 10.10 as 10.1. macOS 11 is requiring tons of patches because many build
> systems expected the macOS major version to remain at 10 forever. Many
> build systems need patches now for ARM support. etc. etc. etc.
>

In this particular case, I believe that the Xcode version, and even the
macOS version, should be counted as just another type of dependency... one
that happens to be required in every single portfile. If these (required)
variables also had the ability to specify version ranges, we could
effectively cap each version of a portfile with a maximum compatible
version of macOS and Xcode, which could be raised as compatibility patches
are added for newer macOSes and Xcodes.

macos_version_compatibility <=10.15.7
(I prefer to use Darwin version)

darwin_version_compatibility <=19
(this portfile would be compatible up to Catalina, but not yet compatible
with macOS 11)

xcode_version_compatibility >=8.2.1 && <=12beta2
(or whatever)

The benefit of this would be as follows. Let's say that we went to a system
of having one portfile for each version of a particular package. Let's also
say that "somepackage" was currently on version 2.4.3, and it is not
compatible with macOS 11. If the upstream authors of "somepackage" were to
add patches that made it compatible with macOS 11 and released it as 2.4.4,
then the new portfile for 2.4.4 would increment darwin_version_compatibility
<=20. However, the portfile for 2.4.3 wouldn't need to be changed at all,
because it wasn't, and never will be, compatible with macOS 11. If a port
maintainer were to come along and backport the macOS 11 patches into the
2.4.3 port, then they could update the 2.4.3 portfile to also be compatible
with darwin <=20. A similar concept applies to Xcode version, ARM support,
etc.

Thus, the idea of keeping a separate portfile for each version of a package
would answer the question of "how you could build historical "Portfiles" on
a newer OS or compiler version", because the answer is: you never would.
Once a "Portfile-2.4.3" was pegged to a maximum of Catalina, it would
potentially never need to be updated again, because it is not compatible
when macOS 11 gets released. Our new "port" command would be able to
indicate to users on a macOS 11 machine that only "Portfile-2.4.4" was
available, because the old historical "Portfile-2.4.3" isn't compatible
with macOS 11.

This is all work that we are here volunteering to do to improve our current
> port collection, but asking everyone to retroactively also fix these
> problems in all older versions would not be acceptable. We don't have a
> sufficient number of contributors to keep our current collection of ports
> up to date, so we certainly don't have enough contributors to keep all
> previous versions of our collection working as well.
>

This is why it would be important in a "new" MacPorts to be able to cap old
versions of a portfile with a maximum compatible version of stuff. ("stuff"
meaning dependencies, macOS version, Xcode version, etc.) This would allow
older versions of portfiles to slowly fade into obsolescence as their
maximum compatible versions of dependencies become more and more
out-of-date. There would be no need to keep retroactively updating them
once they've been capped with maximum compatible versions.

If compatibility patches can be backported, then great, but if they can't
be for any reason, then that's fine too, the old versions of the portfiles
will continue to work for older versions of macOS/Xcode. But in my opinion
that would be a much better situation than what exists now, where there's
only a single Portfile (and a single version of the package) for all users
on all versions of macOS. However, I once again reiterate that I also
believe it would require a massive amount of effort to implement this new
regime.

You might even have to compile each of those copies from source, since each
> copy is going to a different location, and MacPorts binaries are not
> designed to be relocatable. Making our binaries relocatable is something
> I'm interested in but that is also a large challenge.
>

MacPorts binaries aren't designed to be relocatable, because MacPorts, in
my opinion, operates under a very *nix-like philosophy of sharing and
reusing resources. This is also why many macOS apps *are* relocatable:
because they don't share resources with anything else, and thus can play in
their own sandbox. Regardless of where you relocate this sandbox to, there
is very little chance of the app breaking when you move it, because all of
its toys (like the gettext discussed above) move along with it. In many
cases, sharing/reusing resources and relocatability are direct trade-offs
and in opposition to one another. (Obviously, things like @rpath,
@executable_path, @loader_path, etc. were designed to help break this
linkage/trade-off.)

-- 
Jason Liu


On Sun, Oct 4, 2020 at 3:00 AM Ryan Schmidt <ryandesign at macports.org> wrote:

>
>
> > On Oct 3, 2020, at 20:23, Jason Liu wrote:
> >
> >> On Fri, Oct 2, 2020 at 9:00 PM Ryan Schmidt wrote:
> >>
> >> On Oct 2, 2020, at 17:42, Lothar Haeger wrote:
> >>> Instead of creating separate copies of perl for each version, it
> would've probably been smarter to fix the limitation in MacPorts that made
> this workaround necessary, i.e. its inability to maintain and install all
> but the latest version of a port. RPM can do it, DEB can do it, MSI can do
> it, nothing unusual in the grand scheme of package managers in general to
> be able to choose a specific version to install. Just MacPorts did not
> implement it yet and when the necessity arose a seemingly simple workaround
> was chosen instead of solving the underlying problem.
> >>
> >> I have no familiarity with rpm, deb, msi, or other package managers so
> I cannot say whether or how they allow the user to select which version of
> a package to install.
> >
> > Many modern package management systems give both package makers and
> users the ability to specify version numbers, both for packages and for
> dependencies. I call them "modern", because either they have only existed
> for, say, 10 years or less, and were created from the very beginning with
> version-choosing capabilities, or if they have been around for around 20
> years or more, then they gained the ability one of two ways. Either the
> ability was added as part of a complete rewrite of the package management
> system, or it was added as a major new feature.
> >
> > One of the most visible consequences of this ability is demonstrated by
> how dependencies are specified. Giving users the ability to choose which
> version of a package to install could very easily result in dependency
> hell. Thus, to prevent this, dependencies also need to be able to have
> version numbers specified:
> >       • RPM Guide: Specifying the Version of the Dependencies
> >       • ServerFault: RPM: Set Required: somepackage >= 0.5.0 AND
> sompackage < 0.6.0
> > This results in package management systems with spec files that have
> dependencies specified like this:
> >
> > Requires: somepackage >= 0.5.0, somepackage < 0.6.0
>
> Yup, once it has been shown how it would be possible to install multiple
> versions of a port simultaneously and request specific versions, it is
> obvious that dependencies would need to be able to specify versions. What
> specific syntax would be used to do that isn't important at this point.
>
>
> >> I'm speaking of the user being able to specify an arbitrary version. If
> you're instead thinking that the port maintainer would specify a list of
> valid versions or something, that might be more feasible, but still not
> without some of the above problems.
> >
> > APT has the ability to do this. APT is a package management system used
> by Debian, Ubuntu, and other flavors of Linux that use .deb-based packages.
> To repeat what I just linked to, using apt-get, a user can install a
> specific version of a package:
> >
> > apt-get install apache2=2.2.20-1ubuntu1
> >
> > YUM can also do this. YUM is a package management system used by Red Hat
> Enterprise Linux, CentOS, and other flavors of Linux that use RPM-based
> packages. To repeat what I just linked to, using yum:
> >
> > yum install firefox-31.5.3-3.el7_1.x86_64
>
> Where by "this" you again mean the ability to specify a
> previously-available version, not an arbitrary version that was never in a
> portfile.
>
> Those links again do not explain how to overcome the challenges of
> building an old portfile against new dependencies or on new operating
> systems or compilers.
>
>
> > NPM and Yarn, which are PMSes for JavaScript packages, were created with
> package versioning capabilities as a feature right from their very
> beginnings:
>
> npm is one I'm familiar with from an attempt to redesign the MacPorts web
> site several years ago. Its intended use is that you install the modules
> that are needed for your web site project. Each of your projects has its
> own node_modules directory and its own copy of every module. Not only that,
> but each module that you install with npm might have its own dependencies
> which get installed in yet another node_modules directory. This can quickly
> balloon out of control as you get 25 different versions of the same
> dependency. Despite the bloat it might work for that use case, but some npm
> module developers unnecessarily restrict the versions of dependencies
> they're compatible with, meaning that you get an unnecessarily outdated
> version of that dependency, even though a newer version of the dependency
> would work and might fix some bug you're experiencing. And this ballooning
> of dependencies can use a tremendously larger amount of disk space than if
> just a single compatible copy of each dep had been installed. We already
> have users complaining about unnecessary dependencies and unnecessary disk
> usage; this would just make it worse.
>
> Specifying what versions of dependencies are compatible might work in npm
> where modules are mandated to use semver. No such mandate exists for the
> unrestricted global collection of software that could be available in
> MacPorts. If your software currently uses gettext 0.19, how can you know
> whether it will be compatible with gettext 1.0 since that hasn't been
> released yet?
>
> And do you really want to install a separate copy of gettext for each
> dependency that uses it? You might even have to compile each of those
> copies from source, since each copy is going to a different location, and
> MacPorts binaries are not designed to be relocatable. Making our binaries
> relocatable is something I'm interested in but that is also a large
> challenge.
>
> I can't see how the npm strategy could be applied to MacPorts.
>
>
> > port install vlc @5.0.6 +jack +svg +shout
> >
> > and in portfiles, something like this for dependency version ranges
> might make sense:
> >
> > depends_lib port:boost >=1.67.0 +cmake_scripts +python37|python38
> -mpich_devel \
> >             port:openexr >=1.7.4_3,<2.7.1_2
> >
> > (this example also includes a capability on my personal wish-list: the
> ability to specify a dependency to be installed with/without a certain set
> of variants, but without needing to resort to things like using the
> active_variants PortGroup or require_active_variants)
>
> Right, if we've gotten to the point where we can request and install
> arbitrary versions simultaneously to different locations, then it's a
> simple matter to extend that to variants. Until then, it's unworkable.
>
>
> >> As for MacPorts, it's not that we haven't implemented it because we're
> lazy. It's because, besides being an unimaginably large amount of work in
> rearranging our code to do it, I have absolutely no idea how it would be
> accomplished without providing the user with unlimited opportunities to
> create broken combinations of port versions, which would generate an
> unlimited number of bug reports that we would then need to respond to, and
> my goal in MacPorts is to reduce, not increase, the likelihood that users
> would find something broken or need to contact us to help troubleshoot it.
> >
> > I agree, it would be a monumental amount of work. In the worst-case
> scenario, it would require a complete rewrite of MacPorts, including
> redesigning the entire server infrastructure as well as a top-to-bottom
> rewrite of the client software. At a minimum, it would probably require
> significantly changing how the MacPorts "port" command works. For example,
> I have so far submitted two versions of my MaterialX port: version 1.37.1
> and 1.37.2. If I were to do a "port install materialx @1.37.1" (even though
> the current version in MacPorts is 1.37.2), then the MacPorts client
> software would first need to somehow go out onto the internet and fetch the
> portfile for MaterialX version 1.37.1. If you're interested how other PMSes
> accomplish this, I could go into further details.
>
> Hosting Portfiles on a server and downloading them on demand sounds like
> straightforward and simple matter, once the previously mentioned obstacles
> are overcome.
>
>
> > However, I disagree that it would provide unlimited opportunities to
> create broken combinations of port versions. This "unlimited"-ness would be
> prevented by restricting dependencies to a range of version numbers (or
> just a single version number). In reality, it would most likely reduce the
> number of bug reports, since you would be able to cap the maximum version
> of a dependency that a package was compatible with. Right now, MacPorts
> doesn't have that ability.
>
> How would you address the "implicit declaration of function" problem I
> mentioned? A new version of Xcode, 12, has come along and shown us that a
> zillion of our ports, such as php, do not include the headers that declare
> the functions they are using. We must fix this for compatibility with
> future Macs. I am hoping that the developers of php will fix this for
> php72, php73, php74, and I am hoping that I will be able to backport their
> fix to the earlier versions of php that I offer in the php port. I am also
> willing to try to fix this in other ports, but I would not be delighted to
> be asked to fix it for every previous version of those ports.
>
> This is just an example of the type of problem that comes up all the time.
> Yosemite required tons of patches because many build systems misinterpreted
> 10.10 as 10.1. macOS 11 is requiring tons of patches because many build
> systems expected the macOS major version to remain at 10 forever. Many
> build systems need patches now for ARM support. etc. etc. etc. This is all
> work that we are here volunteering to do to improve our current port
> collection, but asking everyone to retroactively also fix these problems in
> all older versions would not be acceptable. We don't have a sufficient
> number of contributors to keep our current collection of ports up to date,
> so we certainly don't have enough contributors to keep all previous
> versions of our collection working as well.
>
>
> Everything we're talking about here seems like much more work than just
> doing what we're doing now to add extra ports for additional versions in
> the specific cases where that's needed.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macports.org/pipermail/macports-dev/attachments/20201004/bee16ede/attachment-0001.htm>