GSoC Proposal

[Clemens Lang]

Hi,

On Thu, Apr 13, 2017 at 12:37:55AM +0200, Mojca Miklavec wrote:
> I was talking to someone from the HB community. Apparently they tried
> very hard to make Travis CI work. One of the major problems they had
> other than the time limit (the builder is relatively slow and jobs get
> killed after 50 minutes, so any builds like Qt would fail of course)
> was the fact that it was impossible to upload the binaries from
> "untrusted pull requests" (not sure what that means, in any case, it
> didn't work for them).

Yeah, that was also the impression I had last time I talked to somebody
from homebrew about their build setup.

> Another option being mentioned was CircleCI (no limitation to upload
> files), but that one has an additional limitation of only 500 free
> minutes per month for OpenSource orgs, so most likely not anywhere
> near enough.

I think in the long run we may not have an other option than providing
the build resources ourselves, but that comes with the responsibility of
containing any effects due to the untrusted input.

> From what I understand there are two different kinds of security
> issues:
> 
> (a) People being able to submit arbitrary code as pull requests
> 
> (b) Most of committers are probably not really inspecting diffs or
> full sources of packages. In theory one can write some originally
> useful opensource tool that we package. Once people start using it,
> some problematic code gets introduced, we happily upgrade the port,
> and both our main builder and any user installing that port would be
> affected.
> 
> We currently don't have a problem with (a) because we don't build pull
> requests, and with Travis we would simply outsource the problem to
> someone else. For (b) we currently don't have any reliable solution,
> but using Travis would not actually help since a maintainer would
> probably eventually merge the code (if it looks reasonable). And then
> the code from official repository would be built on the main builder.

As for (b), we try to run things in sandboxes to limit the attack
surface, but sure, that's an issue we currently have. Ideally, we would
also run 'make install' as non-root user to further reduce the risks.
Realistically speaking, all distributions have the problem of trusting
their upstream code, though.

Our main concern for building PRs are attackers that could try to use
our CI infrastructure to run bots for a botnet or use it to send spam by
sending a pull request with a specially crafted Portfile.


> I'm thinking of an alternative approach that would cover another use
> case (which cannot be solved by Travis CI due to limitations).
> 
> Use cases I have in mind:
> - developer testing a new release of Qt; or a bunch of KDE apps
> - developer trying to build all the 1000+ perl modules once perl5.26
> gets released
> - developer without access to older machines, trying to figure out why
> the build fails on 10.6; having some clue why that could be, but
> trying to avoid doing ten semi-random commits to the main repository
> just to test the hypothesis
> 
> What we *could* do is to set up a few more build slaves on the
> "existing" infrastructure (the build slaves could actually be
> anywhere, people could even volunteer to provide access to their
> machines; but I guess the idea would be to eventually set up a few new
> ones next to the existing slaves). The build master could either run
> in the same instance or separately.

We must take care to not risk the security of the build machines that
build our binary archives. They are a valuable target, so we should
isolate machines that build unreviewed user input, ideally using VMs.

> The idea would be that people with commit access would have to
> manually approve building of a particular pull request (and then that
> pull request would get a green icon of course once the build is
> complete). When a pull request gets updated/modified, the build could
> be tested again, but it would have to be triggered manually again.
> 
> For each manually approved/triggered build the build slave would:
> - clone the repository
> - (?) most likely rebase that on top of master to make sure that the
> latest packages of everything would be used at the time of build
> - determine which ports have been modified
> - maybe run lint as an initial quick test (?)
> - build those ports in a very similar way to what the buildbot does at
> this moment, with a few exceptions:
> 
>   (a) packages from other/existing ports would be fetched from the
> main builders, perhaps even when they are not distributable (we could
> have private copies of binary packages)
>   (b) the resulting packages would be stored between individual builds
> that belong to the same PR, but they would not be uploaded anywhere
> and both sources and binaries (basically anything that has been
> fetched or modified) would be deleted once PR testing is complete

I'm not sure having to review each PR before a build is started is
worthwhile. The whole idea of a PR build bot is that it reduces the
review load on committers, because quick and easy-to-spot feedback is
returned to contributors earlier and faster, and reviewers can spend
their time where it matters most.

> - Some glue code that knows how to communicate between GitHub and
> BuildBot: which jobs to submit, how to report success etc.
> 
> The glue code might need some time to be implemented, but it sounds
> like something in the same direction that you already proposed with
> that "go" bot. To be honest, I currently don't know yet what exactly
> would be needed on that part.

I was under the impression that was the second part proposed by the GSoC
project. First, it's a build under Travis, second is the bot. Ideally,
the bot would be written in a way that it could also trigger other
builds (or we could easily integrate other builders with GitHub, I
believe there's a github PR build option for buildbot as well).


> One of the quick&dirty options would even be to add an additional
> custom field (like "Port list" on https://build.macports.org/builders
> that allows rebuilding an arbitrary port) saying "PR Number" where a
> developer would enter the number of the pull request and that would
> then be built. Another alternative would be to write a simple website
> interface where developers could log in with OAuth and click on a
> build button for open pull requests. Just brainstorming.

Please check whether there are pre-existing solutions for buildbot <->
GitHub integration.


> One of the things I really miss is a website with easy access to build
> summary on per-port basis:
> - https://trac.macports.org/ticket/51995#Statistics
> (This might be something that might even be possible to implement as a
> special view right on the buildbot.)

I always thought that was part of the MacPorts web app Ryan was working
on.


On Sun, Apr 16, 2017 at 03:02:20PM +0000, Zero King wrote:
> I had this idea in mind but chose the one in the proposal because I'd
> like to finish deployment in GSoC and changing the Buildbot
> infrastructure remotely could cause problems. On Travis whatever I do
> I won't break anything hard to recover or leak any secrets but it's
> not so safe on Buildbot. I consider virtualization and snapshots the
> last defence against evil PRs and prefer to have them.

I agree with this. Quick iteration with limited results is more useful
than the best solution that never gets done.

-- 
Clemens