Buildbots Down?

[Ryan Schmidt]

On Jun 22, 2021, at 08:57, Christopher Nielsen wrote:

> Sorry, I meant the buildbot watchers and workers, not the jobs themselves.

The commit that added the p5.32 ports causes ~1800 build requests to be created per worker. Buildbot isn't designed for there to be thousands of pending build requests. Significant slowdowns occur when adding that many build requests (such as no builds being started for many minutes while the buildbot Python process uses 100% CPU doing who knows what). I am not certain, but it seemed like the more build requests there were, the greater the slowdown, as if some sort of SQL cross product among all pending build requests were being computed. To avoid potentially grinding the entire system to a halt, which (like any other forced interruption of the buildbot while builder builds are pending) would be labor-intensive to rectify, I'm only allowing a few workers at a time to process that commit.