Checking for problems before committing

[Joshua Root]

On 2016-3-11 08:41 , David Evans wrote:
> On 3/10/16 12:29 PM, Ryan Schmidt wrote:
>>
>> On Mar 10, 2016, at 12:48 AM, Mojca Miklavec wrote:
>>
>>> On 10 March 2016 at 05:48, Ryan Schmidt wrote:
>>>>
>>>> Obviously nobody is going to commit something they believe is broken, but it does sometimes end up being the case for some subset of users. When it does, and we learn that it has happened, we try to fix it. If everybody had to test everything on a clean system on every version of OS X and every version of Xcode before committing, nobody would ever commit anything because nobody has the time and resources to do all that testing. We do have automated build machines that do build every commit on a clean system with no ports installed with several versions of OS X and the latest version of Xcode for those systems. That automated system did catch this webkit2-gtk build problem on El Capitan,
>>>
>>>
>>> When I was testing wxWidgets, discovered a problem and submitted a
>>> patch, I noticed what they are doing now (which is some light years
>>> more advanced compared to what they did a few years back when most of
>>> the tickets were stuck ignored at their Trac; similar to what happens
>>> in many cases in MacPorts).
>>>
>>> - user submits a patch
>>> - the system checks whether the patch applies cleanly
>>> - the system automatically builds wxWidgets on Windows in 6 different
>>> ways (cygwin, mingw32 with msys, mingw, nmake with VS 14, nmake with
>>> VS 9, msbuild), on Linux in five different ways (one is with clang
>>> 3.5), and on OS X 10.9
>>> - I'm not sure whether wxWidgets does it as well, but it is also
>>> possible to run tests as part of the build
>>>
>>> The developers then only apply the patch if all of the checks mentioned pass.
>>>
>>> The point is that this is all done *in advance* and avoids a lot of
>>> problems. I would love to see something similar being done for patches
>>> submitted to our Trac. Of course they would have to be submitted in a
>>> different way and I'm aware that this is not really trivial to set up.
>>> But it would be extremely helpful.
>>
>> Yes, this would be helpful. I intend to look into doing something like this. However right now and for the next several weeks there are a lot of other more pressing issues I need to be working on for Mac OS Forge.
>
> One approach I have thought about is this:
>
> Establish a "testing" instance of the port repository and have maintainers commit to it.  The buildbots would then build
> from changes to this repo and, upon a successful build, move (copy, merge, whatever) the port to the "live" repository.
> The rsync server would publish from the "live" repository.

I've thought about this too. One issue I can see is that a successful 
port update can still break dependents (see openssl). So unless you 
rebuild all dependents on every commit, you still don't have any sort of 
guarantee that the ports in the live repo work at any given time.

> This is not a detailed proposal but you get the idea and I think could be implemented without too much work. The plus is
> that it would be relatively automated without too many false positives.  Exception cases are ports that "fail" building
> but are considered "good."
>
>    * ports that "fail" because they require non-default dependencies (+quartz is one class, gtk-osx-application for instance)
>    * ports that "fail" because they are meant to (obsolete ports, graveyards)
>    * you can probably think of more
>
> Not a really original idea but at least it would be some filter between maintainer commits and what the user gets

Another approach would be to add a 'try' scheduler to our buildbot config.

<http://docs.buildbot.net/current/manual/cfg-schedulers.html#try-schedulers>

- Josh