Buildbot Performance

[Jason Liu]

On Mon, May 17, 2021 at 7:54 PM Christopher Nielsen <
mascguy at rochester.rr.com> wrote:

>
> Pinning our buildbot VMs to specific NUMA nodes can result in starvation,
> when multiple VMs assigned to a given node are all busy. That would also
> result in underutilization of the other node, if VMs assigned to that are
> idle.
>
> Does that make sense?


What you say does make sense, but I believe that what I am suggesting is
different than what I think you are describing. I'm not suggesting that the
guests get pinned to different NUMA nodes. Instead, I'm suggesting that all
guests get pinned to the same 6 nodes. AFAICT, the starvation situation
that you're describing can occur when some of the guests get pinned to a
different set of NUMA nodes. If all of the guests are pinned to the same
nodes, then theoretically, those guests should behave as if they all have
equal access to the same pool of physical cores. There shouldn't be any
underutilization of any particular node, because the node has been pinned
for all of the guests; this means that the hypervisor should be in control
of scheduling the utilization of the node among all of the guests which
have that node pinned.

-- 
Jason Liu


On Mon, May 17, 2021 at 7:54 PM Christopher Nielsen <
mascguy at rochester.rr.com> wrote:

> Well, my recommendation for our setup, is to avoid pinning.
>
> Why? For several reasons:
> * Out-of-the-box, ESX/ESXi makes a best-effort attempt to schedule all
> vCPUs for a given VM, on a single NUMA node.
> * Even when that’s not possible, the hypervisor schedules VM vCPUs to
> hyperthread pairs.
>
> Pinning our buildbot VMs to specific NUMA nodes can result in starvation,
> when multiple VMs assigned to a given node are all busy. That would also
> result in underutilization of the other node, if VMs assigned to that are
> idle.
>
> Does that make sense?
>
> > On 2021-05-17-M, at 19:34, Jason Liu <jasonliu at umich.edu> wrote:
> >
> > If the guests on a virtual server are exerting a heavy enough load that
> the virtual host is not able to obtain the resources it needs, then the
> entire system's performance, both physical and virtual, can be affected.
> I'm not claiming to be familiar enough with the specifics of the situation
> to claim that this is what's happening, but if it is, then using CPU
> pinning can help. It's basically analogous to the situation where you
> overcommit the memory too much, and the virtual host isn't able to have
> enough memory to do the tasks it needs to.
> >
> > On the virtual servers which I set up for my customers, I have the
> hypervisor set up to automatically pin all but 2 cores. So for example, on
> an 8 core machine, I pin cores 0-5 for all of the VM settings, so that none
> of the guests are able to use cores 6 and 7. This effectively removes 2
> cores from the pool of CPUs available for the guest to use, which means
> that the virtual host will always have those 2 cores available to it at all
> times. This allows my customer to run, on average, 10 guests on the virtual
> server simultaneously, and everything stays performant, even under heavy
> vCPU loads.
> >
> > The other thing I do is that I tell my customers to never overcommit on
> memory. My rule of thumb is that the sum of all simultaneously running
> guests should never exceed the amount of physical RAM minus 4 GB. So, on a
> server with 128 GB of RAM, the sum of the memory allocated to the guests
> running simultaneously should never add up to more than 124 GB of memory.
> >
> >> On Mon, May 17, 2021 at 6:24 PM Ryan Schmidt <ryandesign at macports.org>
> wrote:
> >>
> >>> On May 17, 2021, at 13:13, Jason Liu wrote:
> >>>
> >>> Regarding CPU overcommitment: Are the virtual hosts doing any sort of
> CPU pinning? Many virtualization products have the ability to specify which
> of the pCPU cores a guest is allowed to use. As far as I can remember,
> products like KVM and ESXi can do CPU pinning, while VirtualBox cannot.
> >>
> >> Nope, nothing like that is set up. Is there any reason why we would
> want that?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macports.org/pipermail/macports-dev/attachments/20210517/0aaba915/attachment-0001.htm>