Is it time to start regression testing yet?

[C. Florian Ebeling]

I forgot the link, but it is the one Jeremy posted already:

http://portmill.florianebeling.com/



On Sun, Jun 7, 2009 at 12:21 AM, C. Florian
Ebeling<florian.ebeling at gmail.com> wrote:
> On Sat, Jun 6, 2009 at 9:43 PM, Jordan K. Hubbard<jkh at apple.com> wrote:
>> I know that the word "packaging" is kind of a dirty word in MacPorts-land
>> (perhaps largely due to the fact that certain people just won't stop harping
>> about it :-), so maybe it's time for a new(er) topic in an old conversation:
>>  Testing
>
> This has also been my view, and I have secretly been working on a
> build server I call Portmill, which polls the svn, and builds every
> port that changes. Results get posted to a web app which presents
> them, together with build logs (tails) for the ones that fail. This
> all is in a reasonably usable state just now, and I have spent the day
> adding a feed and links to trac changesets and such convenience stuff.
> The builds happen on my laptop machine right now, and the web app is
> on a small server slice.
>
> Internally this relies heavily on Bryan's MPAB scripts, which I have
> adopted a bit to my needs, e.g. to be able to sync the ports tree and
> bring up a shell inside the chroot to look after the situation for
> debugging.
>
> Wrapping around MPAB is a ruby program 'mpbuildserver', which
> periodically looks into the svn log, pulls changes found there if
> there are any, and builds them inside the chroot environment.
>
> The source is all online visible, and you can find it here:
> http://trac.macports.org/browser/contrib/mpab -- MacPorts Autobuild,
> the chroot environment
> http://github.com/febeling/mpbuildserver -- the actual buider process
> http://github.com/langalex/portmill  -- the web frontend on the results
>
> There is nothing that would stop us from running the mpbuildserver on
> several different machines, potentially with a range of different OS
> versions, and we would get a picture of how well the most recent
> changes work.
>
> Obviously only building the changed port is not the whole story,
> because it does not reveal if dependents break through new versions,
> and this might in fact be a more common case of failure.
>
> But I guess only an minority of maintainers have a set of machines at
> their hands which covers all the OS versions which are theoretically
> supported, and that would be covered by what I have now. So this setup
> offers something valuable already.
>
> Currently the web frontend is quite limited in features and slow
> because of simplistic implementation in certain parts. (The list view
> load the build logs 8). But we can probably get a feel for which
> benefits one can expect from such a system. I hope that the direct
> feedback helps quite a bit, but we will see. Of course I'd be happy to
> relocate everything to a macports or apple server.
>
> Another thing that has to be looked after is ease of deployment. This
> is quite slick for the web app, by virtue of being a
> rails+capistrano+passenger app, but the build
> box/buildserver/buildslave part is a bit rough as of yet. I have
> discussing this with Bryan lately. Bryan also helped with ideas with
> problems I had with registry state, so that the chroot now runs trunk,
> which works better that 1.7.1.
>
> Another milestone I see would be to move the svn polling back onto the
> server, so that it creates a build queue of sorts. That queue could
> then be filled in more than one way, and, depending on the number of
> available build machine, also build dependents of changed ports. Maybe
> also running all combinations of variants.
>
> So, that's Portmill :) Hope you like it and see some use in it. As
> soon as the installation of the build box part is smoothed out a bit I
> would invite volunteers to host build boxes. Or there we could get
> even Apple help and use always-on machines in a data-center.
>
> Cheers,
> Florian
>
>
> --
> Florian Ebeling
> Twitter: febeling
> florian.ebeling at gmail.com
>



-- 
Florian Ebeling
Twitter: febeling
florian.ebeling at gmail.com