Buildbot 1.0

[Mojca Miklavec]

On 20 February 2018 at 00:14, Ryan Schmidt wrote:
> On Feb 19, 2018, at 16:49, Mojca Miklavec wrote:
>
>> Are there any thoughts about migration to buildbot 1.0 (from Ryan in
>> particular)?
>>
>> I wanted to seize the opportunity of some hacking during the MacPorts
>> meeting next month to potentially migrate to buildbot version one, but
>> there is one quite severe drawback, and that's the display of
>> waterfall view (https://github.com/buildbot/buildbot/issues/3884).
>>
>> On the other hand we could at the same time fix a number of other
>> problems, like:
>> - random build order
>> - login with OAuth (using GitHub username)
>> - ...
>>
>> Any thoughts about whether or not to migrate now (= soon) and what
>> issues would make most sense to try to address during hacking
>> sessions?
>
> With the old macOS forge buildbot setup, I tended to use the console view. With the rewritten buildbot setup that we now use, the console view has become useless because it only shows one row for each portwatcher task, with no way to see the results of each individual portbuilder that was spawned from a portwatcher. Therefore I now use the waterfall view.
>
> Buildbot 0.9's waterfall view was terrible when it first came out. I have not followed buildbot's development since then but as far as I know 1.0 hasn't fixed that problem. We could consider writing a new waterfall frontend that meets our needs better.
>
> To be honest I found the entire buildbot 0.9 web interface terrible, incomprehensible, unrecognizable. I consider "buildbot 0.9" (and later) to be a new piece of software, unrelated to buildbot 0.8. Maybe I would come to understand it over time, but I don't know. Maybe someone should fork and continue the development of buildbot 0.8.

While I agree that buildbot 0.9 is nearly unrecognisable compared to 0.8 ...

If we had unlimited resources that could of course be an option, but
I'm pretty sure that nobody from the core team is willing to spend any
considerable effort implementing new features in buildbot 0.8, so
someone else would have to step up and I'm not sure this would happen.
It would make more sense to implement the equivalent of the old
functionality on the new platform.

> As far as I know there is no reason why we could not fix the random build order problem while staying on buildbot 0.8.

I agree with that part.

> My understanding is that buildbot 0.9 and later would indeed help us log in with GitHub OAuth, which would be nice. I originally thought we had a way to work around that with 0.8, but it sounded like it would make anonymous viewing of the web interface impossible, and I'd rather not prevent that. It is unfortunate that GitHub login support was never added to 0.8. I wonder how hard it would be to add it.

Probably not *that* hard, in particular not after the code is already
in 0.9, so many parts could probably be copy-pasted, but I find it
more a question of whether it makes sense to do any such effort. If we
do one day migrate to >= 1.0, that will be wasted effort.

> If I were to work on rewriting the buildbot configuration, and I've thought about it, I would focus on merging the portwatcher and portbuilder into a single thing, as I mentioned on the -infra list in October 2017. I also mentioned in that email that I was writing a longer email about the advantages I believe this consolidation will give us, but I never wrote that longer email. I should probably do that to explain what I'm trying to achieve.

Please do.

> If we succeed at merging portwatcher and portbuilder, that might make the console view useful again, so that we might not care about the 0.9+ waterfall view's deficiencies, though there are still all of the other 0.9+ web interface deficiencies to contend with.

Just to explain: when we started writing the current setup we
initially planned to create a huge number of individual steps based on
"port list". That turned out to be much easier with version 0.9 which
we decided not to use (it was still in development phase etc.). The
current setup with port watcher and builder was merely a workaround
because we had troubles creating new build steps on the fly with
version 0.8.

But on a completely different topic. After talking to lead Buildbot
developer ... his opinion is that creating a "custom view" based on
user demands should be a task that should be possible to accomplish in
something like two days. Probably more for more demanding tasks and
for someone who's still learning.

May I suggest to forget about current limitations of
buildbot-version-whatever and try to start brainstorming of what we
really want to achieve and how our perfect set of views should look
like? Can we prepare some sketches of what we want to see?

Here's something to start with (from my own perspective):

[View Nr. #1: list of commits]

An equivalent of
https://github.com/macports/macports-ports/commits/master with
clickable links to builds from individual commits. Perhaps with a
calendar & clock widget to let the user select the time of when that
commit was noticed by buildbot and added to the build queue. Or maybe
some timeline graph where the user could simply slide on timeline to
get the commits around the selected time slot.

I imagine something like a table with a row looking approximately like this one:

[color or check: failure/success] [timestamp] [short commit message]
[committer] [author] [link to view Nr. #2]


[View Nr. #2: overview of a particular commit or individual forced build]

Top of the page: full info about commit:
- committer name & timestamp
- author name & timestamp
- short & full commit message
- URL to commit on github
- potentially URLs to track tickets and PRs (if applicable)

And then the summary:

            [10.13] [10.12] [10.11] [10.10] [10.9]  [10.8]
[port list] OK      OK      OK      OK      OK      OK
[port A]    OK      OK      FAIL    OK      FAIL    SKIP
[port B]    FAIL    FAIL    FAIL    SKIP    SKIP    SKIP
[port C]    SKIP    OK      OK      FAILDEP FAILDEP FAILDEP


[View Nr. #3: overview of history of builds of a particular port]

Top of the page: all relevant info about the port
- name
- current version
- latest version (from livecheck)
- maintainers (potentially with links to github profiles and/or
existing trac pages)
- maybe link to https://repology.org
- maybe additional info like hyperlinks to dependencies' pages, ...
constructed from separate scripts

And then some kind of a table like this one:

                  [ver][10.13] [10.12] [10.11] [10.10] [10.9]  [10.8]
[commit A] [time] 0.9  OK      OK      OK      OK      FAILDEP SKIP
[commit B] [time] 0.8  OK      OK      OK      OK      FAILDEP SKIP
[force]    [time] 0.8                  OK
[commit C] [time] 0.8  OK      OK      FAIL    FAIL    FAILDEP SKIP
[commit D] [time] 0.7  OK      OK      OK      OK      OK      SKIP

with the ability to go back and forth in time to older and newer
commits. The 'commit A' part would be a hyperlink to view Nr. #2.

"Faildep" in this or previous view would contain name of dependency
and a link to the build page of the failed build where we marked the
port as broken.


We discussed the possibility of letting one of the GSOC students work
on the frontend / javascript & a bit of python (if necessary) to
prepare such views, we could probably get a co-mentor from Buildbot
for that. Now, I'm not sure if we would get a student or if any useful
work would be done during GSOC, but it would certainly help if we can
clearly define what exactly our desires are.

Explanation to Pierre in CC: we have several (tens of) thousand ports.
A single commit can touch multiple ports (in an extreme case like
removing $Id lines during migration from SVN or when adding github
handles of developers it could touch nearly all of them at once), but
usually it is one or a small number of ports. If one commit touches 20
ports, then 20 individual jobs are sent to the scheduler. When a new
OS version gets released, we usually want to build all ports as well.
In the past (old setup) we would build all ports in just a single
build step. That single step took a couple of days to finish and would
generate incomprehensible logs, but at least it did the job. Sometimes
we had to restart it and the already-built ports would be skipped.
With the new setup we try to find workarounds and try to only build a
smaller number of ports at once (my manually specifying which ones to
build) since we are afraid of blocking buildbot. Current setup is much
less effective though (building many ports probably takes three or
four times longer than it did with build-everything-in-a-single-step,
but that's not a showstopper in itself).

Mojca