CI system for PR builds

[Rainer Müller]

On 2018-04-07 19:44, Clemens Lang wrote:
> For these resons, we want to reset the machines to a clean state before
> every build, which we could do with buildbot, but requires some python
> magic that hasn't been written yet.

I think this is mostly written. What is needed is only a little bit of
coding and a lot of testing, which is very time consuming.

So here is the full plan in detail:

Buildbot supports libvirt with "latent buildslaves", even in 0.8.12:

http://docs.buildbot.net/0.8.12/manual/cfg-buildslaves.html#libvirt

When a build is about to be executed on this buildslave, Buildbot
instructs clones the base image to a new (temporary) disk image and then
uses this new disk image to spin up a VM from a clean state.

To run this on ESX or any other hypervisor than qemu, we would need to
implement the disk cloning manually as buildbot only uses qemu-img or cp
on the local machine. Probably we would want to use snapshots instead of
cloning full disks. We can customize this in parts by inheriting from
LibVirtSlave.

This VM will then boot and it is supposed to have a buildslave installed
that will connect back to the buildbot master. From then on, the build
runs as usual. After the build finished, buildbot will shutdown the VM
and discard the modified disk image.

The hardest part is to prepare this VM image. At best, it should be
automated so we can easily recreate it for macOS updates.

1) Install macOS
   Not sure how much can be automated here. For a basic version [1], it
   can just be the instructions that need to be clicked through.
2) Install MacPorts into toolsprefix
3) Install buildslave (and other tools) into the toolsprefix
4) Install configuration for buildslave
   Also set buildslave to start automatically after booting and to
   connect to the master.
5) Install MacPorts into /opt/local

As far as I know, Ryan already has a script that could be used for some
of the steps.

When we have such an, we need to configure a builder in master.cfg that
uses LibVirtSlave. This builder needs a modified portbuilder_factory
that skips the deployment of archives. And finally a changesource that
listens to pull requests via a GitHub WebHook (or just some branch for
testing at first).

At this point the setup would already be working.


If the goal is just to get anything working, you could stop reading
here. But I imagine this setup to be slow, as the VM cannot keep any
persistent state. Therefore, I already made up my mind how this could be
optimized.

As the buildbot has no local packages installed, operations like
checking for the existence of a package over HTTP and then fetching the
file will be slow. I expect this to be one major bottleneck. Our usual
portbuilders have all previously built packages available locally already.

One possible optimization would be to mount distfiles and packages in
the VM over (read-only) NFS from another machine that provides the local
mirror.

mount -t nfs -o ro mirror:.../allpackages \
    /opt/local/var/macports/incoming/verified

Now a 'port install' would just directly use the package from this
directory, which is much faster than doing the usual archivefetch.
However, this expects a flat hierarchy; a single directory containing
all packages. This may impose limitations we have yet to find.

For distfiles, it would be possibile to apply a trick with a union mount
to create a local overlay that allows writing new files:

mount -t nfs -o ro,union mirror:.../distfiles \
    /opt/local/var/macports/distfiles
hdiutil create -o /tmp/distfiles-overlay.dmg -size 20g -fs HFS+J
hdiutil attach -nomount /tmp/distfiles-overlay.dmg  # diskX in output
mount -t hfs -o rw,union /dev/diskXs1 /opt/local/var/macports/distfiles

With this, all file lookups are first attempted in the overlay and if
the file does not exist there, it will be looked up in the layer below,
which is our mirror. New files will always be written to the top layer.
Therefore existing distfiles will be used from the mirror, but new
distfiles can be written to the overlay.


This email got a bit long, but apparently there is an interest in
getting a detailed explanation of how this could be implemented. It is
also good I finally took the time to write these things down, as I have
been thinking about it for a while.

Rainer

[1] For an advanced version of automated macOS installation:
  In the High Sierra installer, the startosinstall script has an
  additional --installpackage option that can point to a .pkg to be
  executed after the installation finished. If, and only if, you also
  disable SIP, you can use the option --volume to install onto a
  different disk than the one in your Mac. Maybe this could be combined
  to fully automate all steps... but that is really advanced.