Installing ports in parallel

[Ryan Schmidt]

On Sep 11, 2012, at 16:12, Chris Jones wrote:

> I still Do not see what you really expect to gain. Installing from source is, in my experience, dominated by the build phase, and this *already* will build ports in parallel (make -j X or similar) where it can. Just run top and you will see this. Your cores will already be fully occupied most of the time, even it's only one instance of port running. At least that is exactly what I see on my machines.

Some ports deliberately have parallel building disabled because it does not work. And those ports that do build in parallel do not build *everything* in parallel. There are some files that can be built in parallel, and some that can only be built serially. Not to mention all phases other than the build phase which are run serially. On larger ports, the configure or destroot phases can take some time. So can fetching.

So I understand the desire to want to install multiple ports in parallel; as I said before, I used this unofficial capability when we had it before. But while it was of some benefit to some power users who knew enough to try to start multiple port installations in separate windows and understood the limitations of the implementation, I'm happier that we no longer have to waste time providing technical support for it.


On Sep 11, 2012, at 15:58, Peng Yu wrote:

> My inclination is that the feature that I request can be useful.

Yes it can, but it doesn't provide so much extra usefulness, above what we already have, that anyone has so far found it worthwhile to invest any extra effort into. Re-enabling the existing unintentional multiple-build capability is not desired because we do not want to increase our support responsibilities again. If you would like to back out the change that added the locking code and compile your own custom version of MacPorts to see how it works in its current state, be our guest, just don't be surprised when you encounter errors, and please don't report them unless you can provide a fix as well. This is the change to back out:

https://trac.macports.org/changeset/70174

Rewriting MacPorts to automatically build entire ports in parallel where possible could be done, but it sounds to me like it would be a rather major rewrite of MacPorts internals, which are already somewhat fragile in their current state. You would be welcome to try to do this rewrite if you're interested, and if you get it working that would be great, but there might be more important features or bug fixes to work on instead.


> As I
> can port using binaries for only a very small fraction of ports.
> Enabling parallel port can truly use all the cores to speed the
> process. Recently, I just migrate from Snow Leopard to Mountain Loin,
> It took almost 8 hours for me to completely install what had been
> previously installed. There should be something done to improve this
> process.

What we have done to improve the process is to introduce binary packages. Binaries are available for many ports today on Snow Leopard and Lion, but as you've discovered, not yet all or in all situations. This is where we should focus our efforts on improving things. The main reason why binaries were not available has been that ports don't indicate what license they're under and therefore we don't know if we can legally distribute them. The solution is for ports to indicate their license. We've made great progress on this already; I think there are only 4000 or so ports left that don't indicate their license. If you can help with determining the license of a port that doesn't indicate it, please file a ticket with a patch in the issue tracker. Another reason for no binaries is if the licenses of the dependencies do not appear to be compatible. The port_binary_distributable.tcl script in our repository can help you identify these situations. Possibly the licenses are specified incorrectly, or possibly a port has a dependency that could be changed (openssl to gnutls, or readline to libedit, etc.) which would make licenses compatible. If you can identify a fix for such a situation, again file a ticket.