Buildbot proposal: combine portwatcher and portbuilder

[Ryan Schmidt]

On Mar 13, 2018, at 09:50, Mojca Miklavec wrote:

> Dear Ryan,
> 
> Please take a look at the discussion summary we just had at the meeting:
> 
> https://trac.macports.org/wiki/Meetings/MacPortsMeeting2018/BuildbotRestructuring


So...

> Ryan's proposal
> 
> Ryan asked us to get rid of the separate portwatcher & portbuilder jobs and re-configure the remaining job to interleave the two types of actions. As we understood it, this was to solve the following problem:
> 
> 	• A commit for portA comes in
> 	• portwatcher for this commit schedules a portbuilder job for portA
> 	• This build takes a long time
> 	• While the build is still running, a new commit for the same port arrives, which queues a portwatcher job
> 	• Another commit for the same port arrives, which queues another portwatcher job
> 	• When the portbuilder job finishes, a useless build is scheduled for portA.

Well, one of the several problems it would solve.

> He proposed the following to solve this:
> 
>  +---------------+
>  |    commit1    |
>  +---------------+
>  |    prepare    |
>  |   resources   |
>  +---------------+
>  |   scheduler1  |
>  +---------------+
>  +---------------+
>  |   port1 dep1  |
>  +---------------+
>  +---------------+
>  |   port1 dep2  |
>  +---------------+
>  +---------------+
>  |    commit2    |
>  +---------------+
>  |    prepare    |
>  |   resources   |
>  +---------------+
>  |   scheduler2  |
>  +---------------+
>  +---------------+
>  |   port1 dep2  |
>  +---------------+
>  +---------------+
>  |     port1     |
>  +---------------+
>  +---------------+
>  |   port2 dep1  |
>  +---------------+
>  +---------------+
>  |   port2 dep2  |
>  +---------------+
>  +---------------+
>  |     port2     |
>  +---------------+

If those boxes for "port2 dep1", "port2 dep2, "port2" are meant to all be separate builds, then that's not what I proposed. I proposed that each build in the combined scheduler would either perform the steps of the current portwatcher, or the steps of the current portbuilder. So what you've shown as separate "port2 dep1" and "port2 dep2" builds or steps, I was proposing would be our current "install deps of port2" portbuilder step, to be followed in the same build by an "install port2" step.


> Unfortunately that introduces the problem with the prepared shared resources (like the portindex, mpbb checkout, ports tree), because we really cannot change the portstree while we have build scheduled in a dependency order that was computed from the old ports tree. This would cause seemingly random and hard-to-debug problems if a follow-up commit changes dependencies of ports.

I don't think this causes any problems; on the contrary, it solves them. If on April 1 I schedule a build of all ports, and it takes a month to finish them all, I don't want it to spend the rest of the month building ports as they were on April 1. I want it to build ports as they are at the time that it gets to the build. If a commit comes in on April 2, I want that to be reflected in the previously-scheduled but still pending builds.

For a specific example on my proposed setup, let's say I:

* force a build of zlib @1.2.11, libpng @1.6.34_1, pngpp @0.2.5_2, and pigz @2.4. (I only specify the port names of course, but those are the versions and revisions that are current at the time I force the build.) This schedules build b1, which "is" a "portwatcher".
* b1(portwatcher) updates the shared resources, and schedules four builds: b2(portbuilder:zlib), b3(portbuilder:libpng), b4(portbuilder:pngpp), b5(portbuilder:pigz).
* b2(portbuilder:zlib) installs dependencies of zlib, installs zlib @1.2.11, cleans up.
* b3(portbuilder:libpng) installs dependencies of libpng, installs libpng @1.6.34_1, cleans up.
* While libpng was building, commit abcd came in that updates zlib to the long-awaited version 2.0! It also revbumps all dependents because zlib 2 has a new library version. This schedules b6(portwatcher) at the end of the queue.
* b6(portwatcher) is selected to run next, because it is a portwatcher and has precedence over all portbuilders. It updates shared resources, schedules b7(portbuilder:zlib) and lots of other builds for the dependents that got revbumped, which includes libpng and pigz.
* b4(portbuilder:pngpp) installs dependencies of pngpp, which include libpng and zlib. Since the shared resources have already been updated by b6(portwatcher), this step of this build is where zlib @2.0 and the revbumped libpng @1.6.34_2 end up getting built. It then installs pngpp @0.2.5_2 and cleans up.
* b5(portbuilder:pigz) installs dependencies of pigz, installs pigz @2.4_1, cleans up. This build was scheduled before commit abcd came in, yet thanks to the shared resources being updated by b6(portwatcher) earlier, it is able to build the latest version of the port available now, and not waste time building an already-superseded version.
* b7(portbuilder:zlib) notices that zlib @2.0 has already been installed and exits early.


> If we were to wait for the steps planned by the first scheduler to finish, we would end up in the same situation as outlined at the beginning, which was our initial approach, but only works with buildbot >= 0.9.
> 
> To avoid the problem outlined by Ryan, we could also just enable build merging on the portwatcher, which would avoid the spurious builds.

I'm aware of the build merging feature but haven't played with it much. I don't know if it would work for that here. We can try it. In our current setup, where portwatcher waits for portbuilders and lots of portwatchers could be pending, merging them could be helpful, if it works.

But I want to continue to pursue my idea, since it would solve more problems than just merging portwatcher builds would.


> Regarding emails: we figured out that it makes no sense to make our
> design decisions based on how annoying it would be to write emails. It
> will probably be easier to write a separate set of scripts to send
> individual emails separately.
> 
> Regarding the width of the waterfall: buildbot one solves that, the
> waterfall is much leaner there :)

...since it removes all the information I want to see...


> I have another request: could you please package buildbot 1.1 in a
> Portfile? Ideally those who have buildbot installed now should get the
> port replaced_by buildbot-0.8 and those who install it from scratch
> should get buildbot version 1.0, but I'm not sure if MacPorts
> currently support such a migration scheme. It would make more sense to
> have buildbot 1.1 named "buildbot" and the old one named
> "buildbot-0.8" though. Perhaps the new one should be called buildbot-1
> after all, at least for a while, I don't know.

Yes I should do that.

https://trac.macports.org/ticket/53006