OpenBLAS binary packages

[Ryan Schmidt]

On May 15, 2019, at 04:01, Artur Szostak wrote:

> This raises the question: how is the MacPorts team mitigating the cost of compiling OpenBLAS over and over again when trying to prepare production binary packages for other ports where OpenBLAS happens to be in the dependency tree?

We keep all* ports installed but deactivated on our buildbot machines. When a new build request comes in, it reactivates the dependencies, which is much quicker than having to rebuild the dependencies from source and also faster than having to download their binaries from a server.

*Since we are running out of space on some of our build machines, we will modify this so that not all ports remain installed there. For example we might uninstall ports that are not dependencies of any other port. See https://trac.macports.org/ticket/57464




On May 15, 2019, at 05:23, Artur Szostak wrote:

> I fear there is also a procedural and policy enforcement issue here. Clearly different people have implemented things in different ways, giving a non-standard and non-uniform look and feel to MacPorts ports. It makes for a poor user experience and the MacPorts community look bad.

As far as I know we don't have a set procedure or policy for this situation of how to handle software that needs to be built differently on different CPUs for performance reasons. The creation of such a policy could certainly be discussed.


> After our 4 years of experience preparing RPM and MacPorts package repositories of our software we have seen that MacPorts packages take an order of magnitude more time to build and cause significantly more problems than RPMs. Mainly because there is a non-trivial probability that some port in the dependency tree will start breaking, and there is no way to pin or encode version ranges for the Portfile dependencies to prevent this.

I'm sorry to hear that your experience with MacPorts hasn't been as good as with other package managers. I don't have experience with package management systems other than MacPorts; I would guess that applies to many of our contributors. So we don't know how others are handling these issues. If you or anyone has suggestions for specific things we should be doing differently, please let us know.

You are correct that MacPorts does not contain a feature that would allow dependencies to be pinned to a version range, and I can't think of a way that such a feature could work. Only one version of a port can be active at a time, so if you support the idea of one port requiring a specific older version of a dependency, you cause problem for all the other ports that wanted the current version of that dependency.

What we do allow is for separate ports to be created for separate versions of the same software, in situations where it is thought that that will be useful; perhaps this is what other package managers are doing as well. For example, we have separate ports for libsdl version 1 and 2, so that ports for software that has been upgraded to support libsdl 2 can use that and ports for older software that requires libsdl 1 can use that. If there's a specific port you're thinking of where we need to similarly offer multiple versions, please let us know.

With regard to a port upgrade causing another port to fail to build, as with any other bug, I don't know what we can do other than to fix the issues as we discover them, and if you've discovered something that fails to build, please let us know by filing a ticket in the issue tracker as usual or by submitting a PR to fix it. We are not mind readers and we often cannot anticipate when a port upgrade would cause problems for another port. Sometimes it is clear from the beginning: it was clear that libsdl 2 was going to be backward-incompatible with libsdl 1, so we made a separate port. Sometimes it is not clear: libpng 1.6 made private certain structure fields that had previously been public, causing some software to fail to build. We could have at that point decided to create a separate libpng 1.4 port for old software; instead, we decided to fix the old ports that were broken by the upgrade, either by upgrading those ports to new libpng-1.6-compatible versions or by patching the software ourselves.

I have a feeling that package management systems on other platforms have more users, because on other platforms the package management system is how libraries and programs used by the OS itself are distributed and updated, so 100% of users of those other platforms will use the package manager, so any problems will be discovered, reported, and fixed quickly. MacPorts, on the other hand, is only used by the fraction of macOS users who have decided to install it, so there is an increased likelihood that you will be the first user to encounter a particular problem vs. other systems with more users. The sooner you report a problem to us, the sooner someone can work on fixing it. Or if you can contribute a fix yourself, that's even better.