CI system for PR builds

[Ryan Schmidt]

On Apr 7, 2018, at 07:02, db wrote:

> On 7 Apr 2018, at 07:47, Mojca Miklavec wrote:
>> No, because that would make the infrastructure that distributes binaries to all our users susceptible to malicious PRs.
> 
> How is reviewing a PR, then letting it build and, if it succeeds, uploading the binary,

We don't want PR builds to upload any binaries anywhere. PRs are unapproved. Only after a PR has been approved and merged to master should a binary be uploaded anywhere. And that's what our buildbot currently does -- it gets notified of the commit to master and does the build.

> if not, reporting the results — different from the current setup, which it seems to me duplicates the process, instead of reusing the infrastructure?
> If a PR fails for all or some systems, then the changes would never make it into the tree, respectively, sparing trac.
> I'd like to understand what is current and what would be the ideal CI for MP and what's needed to get there.

Currently:

When a change is committed, our buildbot system builds the change. If the change relates to ports, it mirrors the distfiles, builds the changed ports using mpbb and uploads the binaries, and regenerates the SQL database that powers ports.php on our web site. If the change relates to base, it builds base, and renders the manpages to HTML and uploads them to the web server. If the change relates to the web site, it validates the files and uploads them to the web server. If the change relates to the guide, it regenerates the guide HTML and uploads it to the web server. Our buildbot system was originally set up in 2011 by macOS forge, and was rebuilt and redesigned when we moved it onto my hardware in late 2016.

When a PR for ports is submitted, Travis CI goes to work building the port using mpbb, and runs port lint, and reports on the lint and build results in the PR. This was set up some time after MacPorts moved to GitHub in late 2016. Travis is a free service for open-source projects. They provide the hardware and the software running on it, and control the ways in which it can be customized, and impose limitations on its use. I was not involved in getting Travis up and running for MacPorts. Travis was selected, rather than reusing any of our existing buildbot infrastructure, presumably because the steps required to accomplish the task on buildbot were unclear or thought to be complicated or time-consuming.


Ideally:

Different developers may have different ideas of what's ideal. Some may wish to keep Travis CI for testing PRs, and just try to incrementally improve it, reduce differences between its behavior and buildbot's, etc. Others may wish to replace it with something that runs on buildbot, so that we can control it completely and aren't subject to the whims and restrictions of the Travis service. I believe I would prefer something running on buildbot, because I dislike having two different systems that behave differently.