Buildbot proposal: combine portwatcher and portbuilder

[Ryan Schmidt]

On Mar 11, 2018, at 12:32, Mojca Miklavec wrote:

> Dear Ryan,
> 
> Conceptually I like the approach a lot.
> 
> *However*, this will generate an incomprehensible amount of emails
> when a new build slave gets added or when some revbumps are done etc.
> In particular from the 10.5/ppc machine. The individual emails will be
> a lot more helpful, but the amount of them ...

Well you'll only get emails for failed builds, of course. Wouldn't you want to know if your ports failed to build on a new version of macOS?

Certainly build failure reports from old OS versions that you already know your ports will fail on are annoying, and we should fix that in base.


> This sounds totally hacky, but one way around that I can see is to
> write a special-purpose mailer. Either a job on the buildbot or a
> special external script that iterates through all the jobs from the
> last hour from all the builders and creates build failure summaries
> for each individual developer (author/committer/maintainer). Maybe
> this could be a build job on the builder master, but I don't know how
> tricky it would get to do it.

Well we can talk about using hourly summary emails instead of individual-build emails or whole-batch emails, but I don't see it as related to my proposal.

> How would you handle duplicate entries in the build list?

The first implementation would not alter how we handle duplicate entries. We currently decide in portwatcher whether to schedule a build of a port, based on whether we have already built that port at that time. Later, I would want to implement https://trac.macports.org/ticket/55078 to cause any duplicate jobs to exit early. Later still, we might look into removing duplicate jobs from the queue at the beginning of a build, assuming buildbot gives us an API to do that.

> Dependency
> order could also become semi-obsolete, but that's probably a price to
> be payed.

Scheduling a batch of builds in dependency order is still an important goal, unaffected by my proposed changes.

> I tried to play with buildNext, but I couldn't figure out how to read
> properties of the build request.

I saw your question to the buildbot mailing list. I haven't tried to use nextBuild yet, but I found an example from the webkit project:

https://trac.webkit.org/changeset/117508/webkit/trunk/Tools/BuildSlaveSupport

And here's somebody else's nextBuild function:

https://gitlab.com/lede/buildbot/blob/8911485994d498e02302c3d474166848b0d894e0/phase1/master.cfg#L249