Package managers and package versioning

Sun Oct 4 07:00:04 UTC 2020

> On Oct 3, 2020, at 20:23, Jason Liu wrote:
> 
>> On Fri, Oct 2, 2020 at 9:00 PM Ryan Schmidt wrote:
>> 
>> On Oct 2, 2020, at 17:42, Lothar Haeger wrote:
>>> Instead of creating separate copies of perl for each version, it would've probably been smarter to fix the limitation in MacPorts that made this workaround necessary, i.e. its inability to maintain and install all but the latest version of a port. RPM can do it, DEB can do it, MSI can do it, nothing unusual in the grand scheme of package managers in general to be able to choose a specific version to install. Just MacPorts did not implement it yet and when the necessity arose a seemingly simple workaround was chosen instead of solving the underlying problem.
>> 
>> I have no familiarity with rpm, deb, msi, or other package managers so I cannot say whether or how they allow the user to select which version of a package to install.
> 
> Many modern package management systems give both package makers and users the ability to specify version numbers, both for packages and for dependencies. I call them "modern", because either they have only existed for, say, 10 years or less, and were created from the very beginning with version-choosing capabilities, or if they have been around for around 20 years or more, then they gained the ability one of two ways. Either the ability was added as part of a complete rewrite of the package management system, or it was added as a major new feature.
> 
> One of the most visible consequences of this ability is demonstrated by how dependencies are specified. Giving users the ability to choose which version of a package to install could very easily result in dependency hell. Thus, to prevent this, dependencies also need to be able to have version numbers specified:
> 	• RPM Guide: Specifying the Version of the Dependencies
> 	• ServerFault: RPM: Set Required: somepackage >= 0.5.0 AND sompackage < 0.6.0
> This results in package management systems with spec files that have dependencies specified like this:
> 
> Requires: somepackage >= 0.5.0, somepackage < 0.6.0

Yup, once it has been shown how it would be possible to install multiple versions of a port simultaneously and request specific versions, it is obvious that dependencies would need to be able to specify versions. What specific syntax would be used to do that isn't important at this point.

>> I'm speaking of the user being able to specify an arbitrary version. If you're instead thinking that the port maintainer would specify a list of valid versions or something, that might be more feasible, but still not without some of the above problems.
> 
> APT has the ability to do this. APT is a package management system used by Debian, Ubuntu, and other flavors of Linux that use .deb-based packages. To repeat what I just linked to, using apt-get, a user can install a specific version of a package:
> 
> apt-get install apache2=2.2.20-1ubuntu1
> 
> YUM can also do this. YUM is a package management system used by Red Hat Enterprise Linux, CentOS, and other flavors of Linux that use RPM-based packages. To repeat what I just linked to, using yum:
> 
> yum install firefox-31.5.3-3.el7_1.x86_64

Where by "this" you again mean the ability to specify a previously-available version, not an arbitrary version that was never in a portfile.

Those links again do not explain how to overcome the challenges of building an old portfile against new dependencies or on new operating systems or compilers.

> NPM and Yarn, which are PMSes for JavaScript packages, were created with package versioning capabilities as a feature right from their very beginnings:

npm is one I'm familiar with from an attempt to redesign the MacPorts web site several years ago. Its intended use is that you install the modules that are needed for your web site project. Each of your projects has its own node_modules directory and its own copy of every module. Not only that, but each module that you install with npm might have its own dependencies which get installed in yet another node_modules directory. This can quickly balloon out of control as you get 25 different versions of the same dependency. Despite the bloat it might work for that use case, but some npm module developers unnecessarily restrict the versions of dependencies they're compatible with, meaning that you get an unnecessarily outdated version of that dependency, even though a newer version of the dependency would work and might fix some bug you're experiencing. And this ballooning of dependencies can use a tremendously larger amount of disk space than if just a single compatible copy of each dep had been installed. We already have users complaining about unnecessary dependencies and unnecessary disk usage; this would just make it worse.

Specifying what versions of dependencies are compatible might work in npm where modules are mandated to use semver. No such mandate exists for the unrestricted global collection of software that could be available in MacPorts. If your software currently uses gettext 0.19, how can you know whether it will be compatible with gettext 1.0 since that hasn't been released yet?

And do you really want to install a separate copy of gettext for each dependency that uses it? You might even have to compile each of those copies from source, since each copy is going to a different location, and MacPorts binaries are not designed to be relocatable. Making our binaries relocatable is something I'm interested in but that is also a large challenge.

I can't see how the npm strategy could be applied to MacPorts. 

> port install vlc @5.0.6 +jack +svg +shout
> 
> and in portfiles, something like this for dependency version ranges might make sense:
> 
> depends_lib port:boost >=1.67.0 +cmake_scripts +python37|python38 -mpich_devel \
>             port:openexr >=1.7.4_3,<2.7.1_2
> 
> (this example also includes a capability on my personal wish-list: the ability to specify a dependency to be installed with/without a certain set of variants, but without needing to resort to things like using the active_variants PortGroup or require_active_variants)

Right, if we've gotten to the point where we can request and install arbitrary versions simultaneously to different locations, then it's a simple matter to extend that to variants. Until then, it's unworkable.

>> As for MacPorts, it's not that we haven't implemented it because we're lazy. It's because, besides being an unimaginably large amount of work in rearranging our code to do it, I have absolutely no idea how it would be accomplished without providing the user with unlimited opportunities to create broken combinations of port versions, which would generate an unlimited number of bug reports that we would then need to respond to, and my goal in MacPorts is to reduce, not increase, the likelihood that users would find something broken or need to contact us to help troubleshoot it.
> 
> I agree, it would be a monumental amount of work. In the worst-case scenario, it would require a complete rewrite of MacPorts, including redesigning the entire server infrastructure as well as a top-to-bottom rewrite of the client software. At a minimum, it would probably require significantly changing how the MacPorts "port" command works. For example, I have so far submitted two versions of my MaterialX port: version 1.37.1 and 1.37.2. If I were to do a "port install materialx @1.37.1" (even though the current version in MacPorts is 1.37.2), then the MacPorts client software would first need to somehow go out onto the internet and fetch the portfile for MaterialX version 1.37.1. If you're interested how other PMSes accomplish this, I could go into further details.

Hosting Portfiles on a server and downloading them on demand sounds like straightforward and simple matter, once the previously mentioned obstacles are overcome.

> However, I disagree that it would provide unlimited opportunities to create broken combinations of port versions. This "unlimited"-ness would be prevented by restricting dependencies to a range of version numbers (or just a single version number). In reality, it would most likely reduce the number of bug reports, since you would be able to cap the maximum version of a dependency that a package was compatible with. Right now, MacPorts doesn't have that ability.

How would you address the "implicit declaration of function" problem I mentioned? A new version of Xcode, 12, has come along and shown us that a zillion of our ports, such as php, do not include the headers that declare the functions they are using. We must fix this for compatibility with future Macs. I am hoping that the developers of php will fix this for php72, php73, php74, and I am hoping that I will be able to backport their fix to the earlier versions of php that I offer in the php port. I am also willing to try to fix this in other ports, but I would not be delighted to be asked to fix it for every previous version of those ports.

This is just an example of the type of problem that comes up all the time. Yosemite required tons of patches because many build systems misinterpreted 10.10 as 10.1. macOS 11 is requiring tons of patches because many build systems expected the macOS major version to remain at 10 forever. Many build systems need patches now for ARM support. etc. etc. etc. This is all work that we are here volunteering to do to improve our current port collection, but asking everyone to retroactively also fix these problems in all older versions would not be acceptable. We don't have a sufficient number of contributors to keep our current collection of ports up to date, so we certainly don't have enough contributors to keep all previous versions of our collection working as well.

Everything we're talking about here seems like much more work than just doing what we're doing now to add extra ports for additional versions in the specific cases where that's needed.