Buildbot Performance

[Ryan Schmidt]

On May 16, 2021, at 09:48, Christopher Nielsen wrote:

> In terms of the ratio of vCPUs to GB of RAM, 1:1 isn’t totally unreasonable. However, we should also reserve 2 GB of RAM for the OS, including the disk cache. So perhaps 6 vCPUs would be a better choice.

MacPorts base hasn't ever considered reserving any RAM for the OS. I cannot confirm that any changes are needed to the formula of how many jobs we start based on how much RAM is available, but if there are, then they should be agreed upon and made in base first; then the buildbot setup can be adjusted accordingly. 


> As for the total physical CPUs available on our Xserves, here’s the rub: While hyperthreading does provide some benefit, best-case it generally only provides 50% more headroom. And sometimes it’s as low as 25%.

I'm aware.


> So if we assume best-case, our Xserve’s only provide the processing power of 12 CPU cores, when accounting for hyperthreading. So even if only two builders are active, we’re already well overcommitted on CPU. And with three or more going, I’d bet the hypervisor is spending more time on scheduling and pre-emption, than actual processing time.

I cannot confirm or deny your claims about the hypervisor.


> By way of comparison, I’m running on a modest 2008-era MacPro, with only eight physical CPU cores… and no hyper threading. Plus the Xeons in my MacPro are one major generation behind the Nehalem-based CPUs on our Xserves. Yet, my port build times are anywhere from 2x to 10x faster than we’re seeing on our builders. (And no, that’s not an exaggeration.)

Are you looking at the time to build just the port, or are you including the time to install all the dependencies? Because on your system, you already have the dependencies installed, or if not, they just need to be downloaded and installed once. On the buildbot, on the other hand, dependencies are deactivated between builds, and are sometimes activated and deactivated many times before the main port is built. This can result in a significant and in my opinion an unnecessarily large amount of time being taken for dealing with dependencies first. I believe this can be improved upon. For example, instead of deactivating *all* active ports between each build and between each dependency, we could develop a way to deactive only the active ports that are not needed by the next build or dependency. This would have the added benefit of reducing disk wear, which has already been a problem before. See https://trac.macports.org/ticket/62621

It is definitely expected that any individual VM will take longer to build if other VMs on the same host are also busy. Reducing the number of CPUs each VM has will just ensure that the builds take longer, even if other VMs on the same host are not busy.


> So we need to do something, as the buildbots simply can’t keep up.

I think they've been keeping up fine, except when a bunch of huge things need to be built, which I find understandable.


> Upgrading them to six-core Xeons would absolutely help, for sure. But I’m quite certain that we could also improve the situation, by reducing the level of CPU overcommitment. And reducing the vCPUs per VM would help, as we simply don’t have the physical CPU power to support eight/VM.

When you say reduce CPU overcommitment, by what means are you referring to, other than reducing CPUs per VM?