GSOC 2019 project: Improvements of Buildbot views (@ MacPorts)

[Ryan Schmidt]

On May 29, 2019, at 13:29, Mojca Miklavec wrote:

> As an intermezzo I want to explain one thing. Initially we had one single job (which could run for days), and it built all the ports in a single job. However one could not infer any information about what worked and what failed without downloading logs first, which could have reached hundreds of megabytes each. We wanted to rewrite it with a custom number of steps, so that a build of 100 ports would contain 100 steps (probably more like a few times hundred; with "clean", "build dependencies", "build port", "upload port", ... for each of the ports individually). The reason for creating port watcher and port builder as separate entities was used as a workaround for lack of functionality in buildbot 0.8, which is no longer the case in the latest buildbot. So maybe it's even time to revise this decision and figure out if there's a better way to organize port watcher/builder into a single builder.

Having a portwatcher and a portbuilder for each OS version causes many problems. I'm all for combining them. We already discussed my proposal for doing that:

https://lists.macports.org/pipermail/macports-dev/2018-March/037577.html


> One problem with current setup is that it may happen that one presses the "rebuild" button and starts building a port on the portbuilder at the same time as portindex is being regenerated, or macports is being updated in the portwatcher. These two actions should not happen at the same time.

This is not a problem currently because each portwatcher build waits for all of its portbuilder builds to finish. It is only currently a problem when there is a power outage or a buildworker crashes. It is annoying when that happens and I have to monitor the builds after everything is back up and usually reschedule some builds that failed because a portindex was being synced during a build, but fortunately it doesn't happen often.


> Another problem is that, say, a perl version increase may result in rebuilding a few thousand ports. Buildbot 0.8 usually choked and became uselessly slow (with builders idling for as long as an hour just to get the next job assigned).

This was only a problem because our buildmaster's disk is slow and our buildbot cache settings were too low. We increased the cache sizes so that the pending builds can all be kept in RAM, so this isn't really a problem anymore, as far as I know.


> Before that rewrite, when everything ran as a single job, we would sometimes schedule builds of all ports. With current setup it's more of a mission impossible to imagine that. I have no idea how buildbot 2.x can handle builds with tens of thousands of steps.

I touched on the problem of occasional huge numbers of builds in my combining proposal above. Builds of huge numbers of ports (whether forced or via a mass commit) could receive a "lesser priority" so that subsequent "normal" commits of one or a small number of ports could receive the normal "higher priority" and "skip the queue" and go first.