Buildbot Performance

[Jason Liu]

On Sun, May 16, 2021 at 3:38 PM Ryan Schmidt <ryandesign at macports.org>
wrote:

>
>
> On May 16, 2021, at 09:48, Christopher Nielsen wrote:
>

>> Upgrading them to six-core Xeons would absolutely help, for sure. But I’m
>> quite certain that we could also improve the situation, by reducing the
>> level of CPU overcommitment. And reducing the vCPUs per VM would help, as
>> we simply don’t have the physical CPU power to support eight/VM.
>>
>
> When you say reduce CPU overcommitment, by what means are you referring
> to, other than reducing CPUs per VM?


Regarding CPU overcommitment: Are the virtual hosts doing any sort of CPU
pinning? Many virtualization products have the ability to specify which of
the pCPU cores a guest is allowed to use. As far as I can remember,
products like KVM and ESXi can do CPU pinning, while VirtualBox cannot.

-- 
Jason Liu


On Sun, May 16, 2021 at 3:38 PM Ryan Schmidt <ryandesign at macports.org>
wrote:

>
>
> On May 16, 2021, at 09:48, Christopher Nielsen wrote:
>
> > In terms of the ratio of vCPUs to GB of RAM, 1:1 isn’t totally
> unreasonable. However, we should also reserve 2 GB of RAM for the OS,
> including the disk cache. So perhaps 6 vCPUs would be a better choice.
>
> MacPorts base hasn't ever considered reserving any RAM for the OS. I
> cannot confirm that any changes are needed to the formula of how many jobs
> we start based on how much RAM is available, but if there are, then they
> should be agreed upon and made in base first; then the buildbot setup can
> be adjusted accordingly.
>
>
> > As for the total physical CPUs available on our Xserves, here’s the rub:
> While hyperthreading does provide some benefit, best-case it generally only
> provides 50% more headroom. And sometimes it’s as low as 25%.
>
> I'm aware.
>
>
> > So if we assume best-case, our Xserve’s only provide the processing
> power of 12 CPU cores, when accounting for hyperthreading. So even if only
> two builders are active, we’re already well overcommitted on CPU. And with
> three or more going, I’d bet the hypervisor is spending more time on
> scheduling and pre-emption, than actual processing time.
>
> I cannot confirm or deny your claims about the hypervisor.
>
>
> > By way of comparison, I’m running on a modest 2008-era MacPro, with only
> eight physical CPU cores… and no hyper threading. Plus the Xeons in my
> MacPro are one major generation behind the Nehalem-based CPUs on our
> Xserves. Yet, my port build times are anywhere from 2x to 10x faster than
> we’re seeing on our builders. (And no, that’s not an exaggeration.)
>
> Are you looking at the time to build just the port, or are you including
> the time to install all the dependencies? Because on your system, you
> already have the dependencies installed, or if not, they just need to be
> downloaded and installed once. On the buildbot, on the other hand,
> dependencies are deactivated between builds, and are sometimes activated
> and deactivated many times before the main port is built. This can result
> in a significant and in my opinion an unnecessarily large amount of time
> being taken for dealing with dependencies first. I believe this can be
> improved upon. For example, instead of deactivating *all* active ports
> between each build and between each dependency, we could develop a way to
> deactive only the active ports that are not needed by the next build or
> dependency. This would have the added benefit of reducing disk wear, which
> has already been a problem before. See
> https://trac.macports.org/ticket/62621
>
> It is definitely expected that any individual VM will take longer to build
> if other VMs on the same host are also busy. Reducing the number of CPUs
> each VM has will just ensure that the builds take longer, even if other VMs
> on the same host are not busy.
>
>
> > So we need to do something, as the buildbots simply can’t keep up.
>
> I think they've been keeping up fine, except when a bunch of huge things
> need to be built, which I find understandable.
>
>
> > Upgrading them to six-core Xeons would absolutely help, for sure. But
> I’m quite certain that we could also improve the situation, by reducing the
> level of CPU overcommitment. And reducing the vCPUs per VM would help, as
> we simply don’t have the physical CPU power to support eight/VM.
>
> When you say reduce CPU overcommitment, by what means are you referring
> to, other than reducing CPUs per VM?
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macports.org/pipermail/macports-dev/attachments/20210517/53618e6f/attachment-0001.htm>