CI system for PR builds

[Ryan Schmidt]

On Apr 7, 2018, at 19:04, Rainer Müller wrote:
> On 2018-04-07 19:44, Clemens Lang wrote:
>> For these resons, we want to reset the machines to a clean state before
>> every build, which we could do with buildbot, but requires some python
>> magic that hasn't been written yet.
> 
> I think this is mostly written. What is needed is only a little bit of
> coding and a lot of testing, which is very time consuming.
> 
> So here is the full plan in detail:

Thanks for writing this up.


> Buildbot supports libvirt with "latent buildslaves", even in 0.8.12:
> 
> http://docs.buildbot.net/0.8.12/manual/cfg-buildslaves.html#libvirt
> 
> When a build is about to be executed on this buildslave, Buildbot
> instructs clones the base image to a new (temporary) disk image and then
> uses this new disk image to spin up a VM from a clean state.
> 
> To run this on ESX or any other hypervisor than qemu, we would need to
> implement the disk cloning manually as buildbot only uses qemu-img or cp
> on the local machine. Probably we would want to use snapshots instead of
> cloning full disks. We can customize this in parts by inheriting from
> LibVirtSlave.
> 
> This VM will then boot and it is supposed to have a buildslave installed
> that will connect back to the buildbot master. From then on, the build
> runs as usual. After the build finished, buildbot will shutdown the VM
> and discard the modified disk image.
> 
> The hardest part is to prepare this VM image. At best, it should be
> automated so we can easily recreate it for macOS updates.
> 
> 1) Install macOS
>   Not sure how much can be automated here. For a basic version [1], it
>   can just be the instructions that need to be clicked through.
> 2) Install MacPorts into toolsprefix
> 3) Install buildslave (and other tools) into the toolsprefix
> 4) Install configuration for buildslave
>   Also set buildslave to start automatically after booting and to
>   connect to the master.
> 5) Install MacPorts into /opt/local

Yes there are tools to automate the OS install, like AutoDMG, which I have played with a few years ago, but there are many nuances and getting everything right is extremely time consuming, given how much time it takes to verify that any change had the desired outcome (since a complete OS install must be performed), not to mention how much disk space is required.

I don't see any reason to automate the creation of these VM images. It should be no more difficult to set them up manually than it was for me to set up the ones we already use for buildbot today.


> As far as I know, Ryan already has a script that could be used for some
> of the steps.

I have this:

https://github.com/ryandesign/macports-infrastructure/blob/master/buildworker-bootstrap.sh

Given a new machine with a fresh install of macOS, Xcode, and the command line tools, the script builds MacPorts from source in the private /opt/bblocal prefix used by the buildbot workers, installs the git port, and adds another ports tree from GitHub containing the ports used for the buildbot infrastructure. The administrator then installs the ports needed for this particular worker. (It's usually just the same port on each worker, except that they use different variants depending on whether the worker will build base or ports or both, and whether libc++ or libstdc++ will be used.)


> When we have such an, we need to configure a builder in master.cfg that
> uses LibVirtSlave. This builder needs a modified portbuilder_factory
> that skips the deployment of archives. And finally a changesource that
> listens to pull requests via a GitHub WebHook (or just some branch for
> testing at first).
> 
> At this point the setup would already be working.
> 
> 
> If the goal is just to get anything working, you could stop reading
> here. But I imagine this setup to be slow, as the VM cannot keep any
> persistent state. Therefore, I already made up my mind how this could be
> optimized.
> 
> As the buildbot has no local packages installed, operations like
> checking for the existence of a package over HTTP and then fetching the
> file will be slow. I expect this to be one major bottleneck. Our usual
> portbuilders have all previously built packages available locally already.
> 
> One possible optimization would be to mount distfiles and packages in
> the VM over (read-only) NFS from another machine that provides the local
> mirror.

I'm not convinced that fetching via HTTP from the private mirror would be significantly slower than fetching via NFS from the private mirror. Either way, the files are coming from the fairly slow RAID card in the master Xserve.


> mount -t nfs -o ro mirror:.../allpackages \
>    /opt/local/var/macports/incoming/verified
> 
> Now a 'port install' would just directly use the package from this
> directory, which is much faster than doing the usual archivefetch.
> However, this expects a flat hierarchy; a single directory containing
> all packages. This may impose limitations we have yet to find.

As you say, we don't have the packages in a flat hierarchy. How would you propose that we create that? Are we talking about yet another parallel hierarchy of hardlinks for our archive deployment script to maintain, like the one we've been talking about in another thread for rearranging the contents of the packages server?


> For distfiles, it would be possibile to apply a trick with a union mount
> to create a local overlay that allows writing new files:
> 
> mount -t nfs -o ro,union mirror:.../distfiles \
>    /opt/local/var/macports/distfiles
> hdiutil create -o /tmp/distfiles-overlay.dmg -size 20g -fs HFS+J
> hdiutil attach -nomount /tmp/distfiles-overlay.dmg  # diskX in output
> mount -t hfs -o rw,union /dev/diskXs1 /opt/local/var/macports/distfiles
> 
> With this, all file lookups are first attempted in the overlay and if
> the file does not exist there, it will be looked up in the layer below,
> which is our mirror. New files will always be written to the top layer.
> Therefore existing distfiles will be used from the mirror, but new
> distfiles can be written to the overlay.

Sure, I imagine that would be possible. But again, I'm not sure that fetching via NFS is faster than fetching from the same server via HTTP.

If we are thinking about having PR test machines running OS X 10.8 or earlier, then we have to consider the HTTPS fetch failure situation again. I don't think we want to mirror distfiles for unapproved PRs to the main distfiles server, so we may need a separate PR distfiles server that could house the files temporarily and delete them later.


> This email got a bit long, but apparently there is an interest in
> getting a detailed explanation of how this could be implemented. It is
> also good I finally took the time to write these things down, as I have
> been thinking about it for a while.
> 
> Rainer
> 
> [1] For an advanced version of automated macOS installation:
>  In the High Sierra installer, the startosinstall script has an
>  additional --installpackage option that can point to a .pkg to be
>  executed after the installation finished. If, and only if, you also
>  disable SIP, you can use the option --volume to install onto a
>  different disk than the one in your Mac. Maybe this could be combined
>  to fully automate all steps... but that is really advanced.