CI system for PR builds

[Rainer Müller]

On 2018-04-08 12:06, Ryan Schmidt wrote:
>> As the buildbot has no local packages installed, operations like
>> checking for the existence of a package over HTTP and then fetching the
>> file will be slow. I expect this to be one major bottleneck. Our usual
>> portbuilders have all previously built packages available locally already.
>>
>> One possible optimization would be to mount distfiles and packages in
>> the VM over (read-only) NFS from another machine that provides the local
>> mirror.
> 
> I'm not convinced that fetching via HTTP from the private mirror would be significantly slower than fetching via NFS from the private mirror. Either way, the files are coming from the fairly slow RAID card in the master Xserve.

The point is that the approach below would not "fetch" the file. It uses
the package directly from NFS, without making a local copy first.

>> mount -t nfs -o ro mirror:.../allpackages \
>>    /opt/local/var/macports/incoming/verified
>>
>> Now a 'port install' would just directly use the package from this
>> directory, which is much faster than doing the usual archivefetch.
>> However, this expects a flat hierarchy; a single directory containing
>> all packages. This may impose limitations we have yet to find.
> 
> As you say, we don't have the packages in a flat hierarchy. How would you propose that we create that? Are we talking about yet another parallel hierarchy of hardlinks for our archive deployment script to maintain, like the one we've been talking about in another thread for rearranging the contents of the packages server?

I am just throwing out ideas. Yes, right now this would require to
create yet another hierarchy. Maybe with changes in base it could reuse
a normal hierarchy.

As I said, that is just for optimization once we have it running.
We can ignore it for now.

Maybe it turns out this is not necessary at all and I am overthinking.
I would be fine with that outcome as well. :-)

> If we are thinking about having PR test machines running OS X 10.8 or earlier, then we have to consider the HTTPS fetch failure situation again. I don't think we want to mirror distfiles for unapproved PRs to the main distfiles server, so we may need a separate PR distfiles server that could house the files temporarily and delete them later.

I would not mirror distfiles for PRs at all.

Testing with just supported versions of macOS should be enough. Anything
on OS X 10.8 and earlier is expected to fail. We need to stop investing
so much effort into legacy systems.

Even just testing with one version would be able to detect the most
common problems, like missing dependencies.

Rainer